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Abstract 

This  investigation  examined  the  different  approaches 
used  in  the  Air  Force  source  selection  process. 

Specifically,  this  research  surveyed  the  opinions  of  source 
selection  personnel  towards  different  aspects  of  each 
approach  in  an  attempt  to  identify  the  approach  best  suited 
for  varying  acquisition  situations.  The  study  also  accessed 
the  adequacy  of  the  Request  for  Proposal  technical 
requirements  definitions  and  determined  the  amount  of  source 
selection  training  personnel  receive  before  participating  in 
a  source  selection. 

The  analysis  was  accomplished  by  sending  a  survey 
questionnaire  to  source  selection  personnel  in  six  Air  Force 
Systems  Command  product  divisions.  The  results  show  that 
the  conventional  approach  is  the  overall  preferred  approach. 
The  results  also  show  that  the  technical  aspects  of  a  weapon 
systems  are  adequately  defined  for  the  contractor  in  the 
Request  for  Proposal.  Finally,  the  results  show  that  89 
percent  of  source  selection  personnel  do  not  receive 
training  prior  to  participating  in  their  first  source 
selection. 


WEAPON  SYSTEM  SOURCE  SELECTION: AN  ASSESSMENT  OF  AIR  FORCE 
SOURCE  SELECTION  APPROACHES 


I.  The  Research  Problem 


Introduction 

In  recent  years,  the  United  States  Air  Force  has  come 
under  close  scrutiny  concerning  how  it  is  spending  the 
taxpayers'  money.  Congress,  the  news  media,  and  the 
Department  of  Defense  are  all  monitoring  the  procurement 
process,  trying  to  insure  that  the  Government  purchases 
quality  products  at  fair  and  reasonable  prices.  The 
procurement  of  major  weapon  systems  and  subsystems  is  of 
particular  interest  since  large  amounts  of  money  are 
involved. 

Background 

The  acquisition  of  major  weapon  systems  and  subsystems 
in  the  United  States  Air  Force  is  a  very  important  and 
complicated  process.  The  acquisition  process  involves  large 
expenditures  of  taxpayers'  money  in  procuring  the  equipment 
and  services  our  country  requires  for  defense.  The  basic 
principles  underlying  the  acquisition  process  have  changed 
and  grown  over  the  years.  The  process  is  continually  being 
modified  by  legislative  and  administrative  actions.  The 
modifications  are  attempts  to  develop  a  system  providing 
safeguards  "against  graft,  favoritism,  questionable  ethics, 
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collusion,  and  inefficiency,  and  to  protect  the  integrity  of 
the  competitive  and  public  bidding  system"  (4:146).  As  of 
April  1,  1984,  all  new  government  acquisitions  must  follow 
the  guidelines  outlined  in  the  Federal  Acquisition 
Regulation  (FAR) .  The  FAR  has  been  in  development  since  the 
1972  Commission  on  Government  Procurement  Report  called  for 
a  simplification  of  the  procurement  regulations  and  a 
reduction  in  the  regulatory  differences  among  government 
agencies.  The  FAR  combines  the  main  elements  of  the 
Department  of  Defense  (DOD)  Defense  Acquisition  Regulation 
(DAR) ,  the  NASA  Procurement  Regulations,  and  the  GSA  Federal 
Procurement  Regulations  which  applied  to  non-DOD  and  non- 
NASA  procurements  (5:25-26) . 

Under  the  FAR,  the  two  principle  acquisition  techniques 
are  formal  advertising  and  negotiations.  Formal  advertising 
is  the  traditional,  most  frequently  used  method  of  doing 
business  with  private  firms.  The  FAR  lists  17  exceptions  to 
formal  advertising  as  conditions  for  using  negotiations. 

The  formal  advertising  method  is  a  technique  which  minimizes 
the  individual  judgment  factor  in  awarding  contracts  by 
having  sealed  bids  opened  in  public  and  by  awarding  the 
contract  to  the  lowest  responsive  bidder  (14:55)  .  Off-the- 
shelf  equipment  purchases  and  service  contracts  are  the  most 
common  applications  of  the  formal  advertising  technique. 

Government  acquisitions  through  formal  advertising  are 
not  always  practical,  especially  in  the  research  and 


development  and  the  DOD  weapon  system  acquisition  areas. 

Over  the  past  25  years,  the  United  States  has  made  great 
advances  in  the  high  technology  fields  of  electronics, 
aerospace,  computers,  and  communications.  The  DOD 
determined  that  the  contractors'  bids  must  be  evaluated  on 
technical  and  management  capabilities  as  well  as  cost. 
Negotiations  allow  the  DOD  to  procure  the  best  possible 
systems  to  meet  our  national  defense  objectives  at  a 
reasonable  cost  while  giving  the  Air  Force  greater 
flexibility  in  the  selection  of  the  best  contractor  to  do 
the  required  work  (2:4). 

During  July,  1984,  Congress  passed  the  Deficit 
Reduction  Act.  As  a  result  of  this  act,  the  Office  of 
Management  and  Budget  (OMB) ,  which  is  responsible  for 
setting  acquisition  policy  in  the  executive  branch  of  the 
government,  issued  new  acquisition  guidance  to  all  Federal 
agencies.  The  Deficit  Reduction  Act  is  intended  to  increase 
the  competition  for  the  government's  business  by  putting 
formal  advertising,  renamed  sealed  bids,  and  negotiations, 
renamed  competitive  proposals,  on  an  equal  basis  for 
awarding  contracts.  The  act,  which  went  into  effect  in 
April  1985,  changed  the  conditions  for  conducting  source 
selections.  Two  methods  of  choosing  a  contractor  via 
competitive  proposals,  the  conventional  approach  and  the 
four  step  approach,  have  been  carefully  designed  to  minimize 
the  risk  of  lawsuits  by  using  procedures  that  give  all 
contractors  an  equal  opportunity  to  obtain  the  contract. 


In  order  to  achieve  equal  opportunity,  every  negotiated 
acquisition  must  pass  through  a  rigorous  evaluation 
procedure  known  as  the  source  selection  process  .  According 
to  Department  of  Defense  Directive  (DODD)  4105.62,  the 
source  selection  process  must  satisfy  three  major 
objectives.  These  objectives  are  to: 

1.  select  the  source  whose  proposal  has  the  highest 
degree  of  realism  and  credibility  and  whose 
performance  is  expected  to  best  meet  Government 
objectives  at  an  affordable  cost; 

2.  assure  impartial,  equitable,  and  comprehensive 
evaluation  of  competitors'  proposals  and  related 
capabilities;  and 

3.  maximize  efficiency  and  minimize  complexity  of 
solicitation,  evaluation,  and  the  selection 
decision. [23:2] 

DODD  4105.62  further  states  that  each  "DOD  component 
shall  develop,  and  consistently  apply,  procedures  which 
create  the  environment  for  an  impartial,  balanced  and 
realistic  appraisal  of  all  proposals  submitted”  (23:2).  In 
response  to  this  directive,  the  Air  Force  has  developed  its 
own  source  selection  policy  and  procedures  in  Air  Force 
Regulation  (AFR)  70-15. 

The  major  objective  of  Air  Force  source  selection  is 
stated  as  follows. 

The  prime  objective  of  proposal  evaluation  and 
source  selection  is  to  assure  impartial, 
equitable,  and  comprehensive  evaluations  of 
competitive  proposals  and  to  assure  selection 
of  that  source  whose  proposal,  as  submitted, 
offers  optimum  satisfaction  of  the  Government's 
objectives  including  cost,  schedule,  and 
performance.  [21:1-1] 


Air  Force  Systems  Command  uses  two  primary  approaches 
in  order  to  satisfy  source  selection  requirements.  They  are 
the  conventional  approach  and  the  four  step  approach.  Both 
approaches  use  the  Source  Selection  Evaluation  Board  (SSEB) 
to  evaluate  the  details  of  each  contractor's  proposal.  The 
approaches  differ  in  in  the  way  they  address  the  proposal 
contents. 

The  conventional  approach  uses  the  SSEB  to  evaluate  all 
technical  and  cost  aspects  of  proposals  together.  The  SSEB 
identifies  any  proposal  deficiencies  to  the  appropriate 
contractor  and  assures  that  all  proposals  meet  minimum 
Government  requirements.  The  SSEB  negotiates  with  all  the 
contractors  and  then  forwards  each  contractor's  "best  and 
final  offer"  to  the  Source  Selection  Advisory  Council  (SSAC) 
where  the  documentation  and  recommendations  of  the  SSEB  are 
reviewed.  After  the  SSAC  review,  all  recommendations  and 
proposals  are  given  to  the  Source  Selection  Authority  (SSA) 
for  a  final  decision  on  contract  award. 

The  four  step  approach  requires  the  contractors  to 
submit  the  technical  and  cost  proposals  separately.  The 
technical  proposals  are  evaluated  first.  During  this 
evaluation,  the  contractors  are  contacted  only  if 
clarification  is  needed  on  technical  issues.  Next,  the  cost 
proposals  are  submitted  for  evaluation.  After  both 
evaluations  are  completed,  a  single  contractor  is  chosen  to 
enter  into  final  negotiation  with  the  Government.  If  the 


Government  cannot  negotiate  an  acceptable  contract  with  the 
chosen  contractor  before  a  mutually  established  deadline, 
the  Government  has  the  option  to  cease  negotiations  with 
that  contractor  and  pick  another  contractor  for 
negotiations. 

Criticisms  of  the  conventional  approach  led  to  the 
development  of  the  four  step  approach.  The  four  step 
approach  was  designed  to  shorten  the  time  required  for  the 
source  selection  process  and  also  to  reduce  the  number  of 
people  required  to  research  and  evaluate  the  contractors 
proposals. 

Problem  Statement 

Since  1978,  when  the  four  step  approach  was  first 
authorized  for  Air  Force  use  in  negotiated  procurements, 
very  few  source  selections  have  used  it.  Source  selection 
personnel  may  not  use  the  four  step  approach  because  they 
are  biased  towards  the  conventional  approach  or  one  of  its 
streamlined  modified  forms.  This  bias  could  be  due  to  the 
level  of  military  and  civilian  education  of  source  selection 
personnel,  previous  source  selection  experience,  or  a  lack 
of  source  selection  training.  Source  selection  personnel 
also  may  not  use  the  four  step  approach  because  the  weapon 
system  or  subsystem  could  have  underlying  characteristics 
which  make  it  unsuitable  for  four  step  evaluation 
techniques.  The  purpose  of  our  research  is  to  determine  why 
Air  Force  Systems  Command  source  selection  personnel  choose 

one  source  selection  approach  over  another. 
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Research  Objectives 

1.  Determine  if  an  adequate  data  base  (source 
selection  records  and  personnel)  exists  and  is  accessible 
for  research. 

2.  Match  a  completed  four  step  acquisition  with  an 
equivalent  conventional  acquisition  and  determine  if 
contract  relationships  can  be  used  to  characterize  a 
specific  approach's  use. 

3.  Determine  if  source  selection  approach  familiarity 
plays  a  role  in  which  source  selection  approaches  Air  Force 
Systems  Command  personnel  consider  during  the  Source 
Selection  Plan  generation. 

4.  Determine  if  source  selection  personnel  prefer  a 
specific  source  selection  approach  based  upon  an 
acquisition's  characteristics. 

5.  Determine  if  source  selection  personnel  have 
preconceived  notions  about  the  time  and  manpower 
requirements  for  a  given  source  selection  approach. 

6.  Determine  if  the  source  selection  process  is  being 
used  as  a  means  to  further  define  weapon  system 
requirements. 

7.  Determine  the  perceptions  of  source  selection 
personnel  towards  the  major  identifying  characteristics  of 
each  source  selection  approach. 

8.  Determine  what  training  source  selection  personnel 
receive  in  the  area  of  source  selection. 


Scope  and  Limitations  of  Research 

For  fiscal  year  1986,  40  billion  dollars  (over  36  per 
cent  of  the  Air  Force  budget)  is  projected  to  go  to  Air 
Force  Systems  Command  (AFSC)  for  the  acquisition  of  major 
weapon  systems  and  subsystems  (12).  Of  AFSC's  share  of  the 
budget,  90  per  cent  of  these  funds  will  be  spent  by  AFSC 
Product  Divisions  (12).  Because  the  bulk  of  the  Air  Force's 
acquisition  funds  are  spent  in  AFSC  Product  Divisions,  only 
source  selection  records  and  personnel  from  AFSC  will  be 
investigated  during  this  research  project.  Procurements 
conducted  outside  of  AFSC  Product  Divisions  will  not  be 


considered. 


II.  Literature  Review 


Introduction 

Procurement  of  major  weapon  systems  and  subsystems  in 
the  United  States  Air  Force  is  a  very  complex  process.  Each 
procurement  involves  coordination  among  many  different  Air 
Force  agencies.  It  also  requires  the  development  of  the 
draft  system  specification  requirements,  a  task  that  can 
take  considerable  time  and  effort.  Once  the  Air  Force  is 
able  to  specify  the  desired  system  requirements  and  define 
proposal  evaluation  criteria,  potential  contractors  are 
notified  by  means  of  a  Request  for  Proposal  (RFP) .  The  RFP 
conveys  a  complete  description  of  the  work  to  be  performed. 
In  response  to  the  RFP,  contractors  interested  in  building 
the  system  prepare  proposals  for  submission  to  the  Air 
Force.  The  proposals  allow  the  Government  to  determine  the 
contractor's  capability  and  the  price  of  his  efforts. 

The  process  of  evaluating  proposals  is  a  critical  step 
in  the  Air  Force  procurement  process.  "There  are  a  number 
of  qualified  sources  in  the  United  States  that  have  the 
prerequisite  experience,  capabilities,  and  facilities  for 
the  development  and  production  of  major  systems  and 
subsystems"  (1:2).  The  potential  of  having  a  large  number 
of  proposals  for  a  given  system  exists;  therefore,  the  Air 
Force  developed  a  process  of  evaluating  proposals  called  the 
source  selection  process. 
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The  source  selection  process  gives  all  interested 
contractors  the  right  to  compete  for  any  program  involving 
the  expenditure  of  public  funds.  All  proposals  received  are 
considered,  and  the  source  selection  process  determines  the 
company  or  companies  best  able  to  build  a  system.  The 
primary  objectives  of  the  source  selection  process  are: 

1.  maximize  competition; 

2.  minimize  the  complexity  of  the  solicitation; 

3.  ensure  impartial  and  comprehensive 
evaluation  of  an  offeror's  proposals;  and 

4.  ensure  selection  of  the  source  whose 
proposal  has  the  highest  degree  of  realism  and 
whose  performance  is  expected  to  best  meet 
stated  Government  requirements.  [25:16937] 

At  present,  two  main  source  selection  approaches  are 

used  in  the  source  selection  process.  They  are  the 

conventional  approach  and  the  four  step  approach.  The 

conventional  approach  also  has  two  modified  forms  which  have 

become  known  as  streamlined  approaches.  The  purpose  of  this 

section  is  to  present  an  overview  of  the  characteristics 

common  to  all  approaches  and  then  summarize  each  approach. 

Characteristics  Common  To  All  Approaches 

The  Air  Force  initiates  the  source  selection  process  by 
generating  a  Source  Selection  Plan  (SSP)  .  The  SSP  specifies 
the  course  of  action  to  be  followed  throughout  the  source 
selection  process  for  a  given  acquisition.  It  also 
establishes  the  standards  and  criteria  to  be  used  in 
evaluating  contractors'  proposals  by  identifying  broad 


general  areas  of  the  program  such  as  logistics,  management, 
engineering,  etc  and  ranking  the  areas  in  order  of  relative 
importance  to  the  program  (2:21)  . 

Source  Selection  Plan  Preparation.  The  Source 
Selection  Plan  (SSP)  "documents  the  criteria  to  be  used  for 
the  evaluation  and  selection"  process  (2:22).  A  SSP  must  be 
written  by  the  government  for  all  acquisitions  above  $1M. 

The  SSP  is  prepared  by  the  government  agency  in  charge  of 
the  source  selection. 

The  parts  of  the  SSP  are: 

1)  a  description  of  the  (source  selection) 

organization  structure; 

2)  proposal  presolicitation  activities; 

3)  a  summary  of  the  acquisition  strategy; 

4)  a  statement  of  the  proposed  evaluation 

factors  and  their  relative  importance; 

5)  a  description  of  the  evaluation  process, 

methodology,  and  techniques  to  be  used;  and 

6)  a  schedule  of  significant  milestones. 

[25:16941] 

This  plan  controls  the  entire  evaluation  and  selection 
process  and  must  be  approved  by  the  Source  Selection 
Authority  (SSA) .  The  SSA  is  "an  official  designated  to 
direct  the  source  selection  process,  select  the  sources,  and 
announce  the  contract  award"  (7:636-637).  Normally,  the  SSA 
is  the  Secretary  of  the  Air  Force  or  his  designated 
representative. 


Establishing  the  Evaluation  Criteria.  Establishment  of 
the  evaluation  critaria  is  a  complex,  critical  task  and  is 
tne  heart  of  the  selection/evaluation  process  (14:202).  The 
definition  of  evaluation  criteria  usually  occurs  during  the 
SSP  preparation  and  becomes  a  part  of  the  SSP  and  the  RFP . 
The  evaluation  criteria  break  down  the  two  basic 
considerations  of  a  proposal  that  are  analyzed  in  the 
selection  process.  One  consideration  describes  the  product 
or  service  being  procured,  and  the  other  consideration 
addresses  the  ability  of  a  company  to  furnish  the  product  or 
service  (2:28) .  There  are  no  set  rules  for  establishing  the 
criteria.  Factors  generally  included  in  the  criteria  are 
cost  realism,  technical  excellence,  management  capability, 
personnel  qualifications,  experience,  past  performance,  and 
schedule  (25:16937).  Cost  must  always  be  included  as  an 
evaluation  factor  but  is  not  necessarily  the  driving  factor 
in  a  source  selection. 

The  standards  for  rating  and  grading  the  contractor 
proposals  against  the  evaluation  criteria  are  developed 
during  SSP  generation.  "A  standard  is  defined  as  an 
acknowledged  measure  of  comparison.  A  standard,  in  order  to 
be  useful,  must  be  measurable"  (19:27).  A  contractor's  work 
efforts  are  measured  by  comparison  to  the  established 
standards.  The  purpose  of  the  comparison  is  to  ascertain 
the  quality  of  the  contractor's  work.  Contractor  proposals 
are  evaluated  against  the  SSP  established  standards  and  not 
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against  each  other.  The  standards  are  never  released  to  the 
contractors.  The  evaluation  criteria  and  standards  provide 
the  framework  for  conducting  the  evaluation  of  the 
proposals.  The  establishment  of  standards  and  subsequent 
evaluation  of  contractor  proposals  against  them  is  a  process 
designed  to  provide  "consistency,  objectivity,  and 
comparability  between  proposals"  (19:27). 

Request  for  Proposal  Preparation.  The  RFP  is  the 
official  government  document  inviting  private  firms  to 
submit  proposals  for  a  contract  to  the  government.  RFP 
guidelines  are  set  forth  in  AFSC  Pamphlet  70-4.  The  RFP 
contains  a  model  contract  -  the  "key  communication  to 
potential  contractors  on  exactly  what,  how,  and  when  the 
Government  needs  to  buy"  (14:205).  According  to  the  Weapons 
Systems  Acquisition  Guide  by  Major  James  Huffman  and  others 
(14:58-59),  a  typical  model  contract  is  broken  down  into  the 
following  sections: 

A.  Contract  Form  -  Usually  Standard  Form  33,  Part  1 
(Solicitation,  Offer,  and  Award,  and  acknowledgment  of 
Agreements) . 

B.  Supplies/Services  and  Prices  -  A  brief  description 
of  the  items  being  acquired.  If  a  specification  has  been 
approved,  the  specification  number  would  be  cited  in  this 
section.  The  contractor  must  build  the  equipment  in 
accordance  with  this  specification. 
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C.  Description/Specifications  -  When  a  brief 
description  in  section  B  is  not  sufficient  to  permit  a  full 
competition,  this  section  is  included  to  provide  more 
detail.  Government  performance  requirements  or  Statements 
of  work  (SOW)  may  be  printed  in  their  entirety  in  this 
section  or  they  may  be  included  as  attachments  in  Section  J 

D.  Packaging  and  Marking  -  Packaging  and  marking 
requirements,  if  any. 

E.  Inspection  and  Acceptance  -  Describes  the  place  of 
inspection  and  place  of  acceptance. 

F.  Deliveries  or  Performance  -  Describes  when  and 
where  items  are  to  be  delivered. 

G.  Contract  Administration  Data  -  Instructs  paying 
offices  and  administrative  contracting  offices. 

H.  Special  Provisions  -  Includes  any  special  (not 
mandatory  by  law)  clauses  that  will  apply  to  the 
acquisition. 

I.  General  Provisions  -  General  contract  provisions 
required  by  law. 

J.  List  of  Documents,  Exhibits,  and  other 
Attachments  -  The  documents,  exhibits,  and  attachments  that 
make  up  the  request  for  proposal.  A  list  of  deliverable 
data,  the  Contract  Data  Requirements  List  (CDRL)  is  usually 
found  in  this  section. 

K.  Representations,  Certifications,  and  Other 
Statements  of  Offeror  -  An  extensive  list  of  statements  the 


contractor  must  certify  to.  He  must  fill  out  the 
certifications  and  return  the  package  with  his  proposal. 

L.  Instructions,  Conditions,  and  Notices  to  Offerors  - 
Additional  instructions  such  as  the  type  of  contract 
desired,  permission  to  submit  alternate  offers,  and 
directions  for  obtaining  copies  of  any  documents.  This 
section  can  include  any  instructions  the  Government  feels  is 
necessary. 

M.  Factors  other  than  price  which  will  be  given 
paramount  consideration  in  the  awarding  of  the  contract. 

The  RFP  lists  the  evaluation  criteria  in  order  of 
importance  to  the  government.  The  criteria  become  the 
guidelines  for  the  contractors  in  preparing  their  proposals. 

If  required  in  the  SSP,  a  draft  RFP  is  released  to 
industry  prior  to  the  formal  release  of  the  RFP.  Draft  RFPs 
allow  the  government  and  industry  to  communicate  on  the  RFP 
package  as  a  whole  and  to  clarify  any  ambiguous  information 
prior  to  the  release  of  the  formal  RFP  (14:233).  In  DOD,  a 
review  of  all  RFPs  is  conducted  by  a  Solicitation  Review 
Panel  (SRP) ,  also  called  a  "murder  board,"  prior  to  release 
of  an  RFP  to  industry. 

The  SRP  makes  sure  each  RFP  provides  a  basis 
for  a  sound  viable  contract;  reflects  desired 
program  objectives;  conforms  to  current 
acquisition,  contracting,  manufacturing,  and 
Quality  Assurance  policy;  outlines  in  clear, 
concise  terms  exactly  what  the  Government 
intends  to  buy;  and  eliminates  excessive  or 
non-essential  contractual  requirements.  (14:203] 


The  source  selection  characteristics  described  in  the 


preceding  section  are  common  to  all  major  weapon  system  and 
subsystem  source  selections.  The  SSP  guides  the  entire 
acquisition  process  with  established  evaluation  criteria  for 
all  contractor  proposals  submitted  in  response  to  the 
government's  RFP. 

During  the  SSP  development,  the  Air  Force  must  make  a 
decision  on  which  source  selection  approach  to  use  in  order 
to  evaluate  the  RFPs.  The  Air  Force  currently  has  two  major 
source  selection  approaches  available.  The  following  is  an 
overview  of  each  approach  and  their  modified  forms. 


Overview  of  the  Conventional  Approach 

Historical  Application  of  the  Conventional  Approach. 
Acquisitions  made  under  the  conventional  approach  in 
accordance  with  Air  Force  Regulation  70-15  require  a  great 
deal  of  time  and  manpower  (2:10)  .  A  review  of  ASD  source 
selection  records  indicates  that  during  the  period  from  16 
January  1981  until  2  March  1934,  Aeronautical  Systems 
Division  (ASD)  completed  six  major  source  selections 
utilizing  the  conventional  approach.  The  average  time 
required  for  each  ASD  source  selection  from  RFP  release 
until  contract  award  was  182  days.  The  average  number  of 
Source  Selection  Evaluation  Board  (SSEB)  members  was  276 
people.  Table  2.1  summarizes  the  six  ASD  conventional 
source  selections  from  16  January  1981  to  2  March  1984. 


TABLE  2.1 


ASD  Conventional  Source  Selections* 


ASD 

WEAPON 

SYSTEM 

DAYS  FROM  RFP 
RELEASE  UNTIL 
CONTRACT  AWARD 

NUMBER  OF  SOURCE 
SELECTION  EVALUATION 
BOARD  MEMBERS 

Alternate  Fighter 
Engine 

177 

246 

Operational  Support 
Aircraft 

126 

146 

Transport  C-17 
Aircraft 

138 

560 

Special  Airlift 
Mission  Aircraft 

207 

360 

Next  Generation 
Trainer 

186 

186 

European  Distribution 

Support  Aircraft 

209 

160 

AVERAGE  182  days  276  people 

*  Compiled  from  ASD  source  selection  records 

After  the  SSP  is  completed,  the  Government  issues  the  RFP 
which  asks  industry  to  submit  contract  proposals  (14:205) . 
The  RFP  contains  a  specific  cut-off  date  for  the  submission 
of  proposals.  Late  proposals  are  rejected.  After  the  cut¬ 
off  date,  the  Source  Selection  Evaluation  Board  (SSEB) 
evaluates  the  proposals  against  the  standards  associated 
with  each  evaluation  criteria  factor.  After  an  initial 
evaluation  of  all  the  proposals,  a  "competitive  range"  is 
defined  by  the  Contracting  Officer  in  charge  of  the  source 


selection.  The  competitive  range  defines  both  the  minimum 
and  maximum  requirements  in  each  functional  area  that  the 
government  expects  to  see  in  the  final  contract.  All 
proposals  within  the  competitive  range  proceed  into  parallel 
negotiations. 

During  the  parallel  negotiations  phase,  the  Contracting 
Officer  negotiates  definitive  contracts  with  all  the 
companies  within  the  competitive  range.  At  the  same  time, 
the  evaluation  board  requests  clarification  of  any  ambiguous 
proposal  items  and  correction  of  proposal  deficiencies  from 
the  companies  in  the  competitive  range.  The  negotiation 
phase  is  concluded  with  the  contractor's  submission  of  a 
"best  and  final  offer"  and  a  signed  contract  to  the 
government  agency  (14:204). 

The  "best  and  final  offer"  of  each  contractor  is 
evaluated  by  the  SSEB  against  previously  established 
^valuation  criteria.  After  the  evaluation,  the  SSEB 
forwards  its  recommendations  to  the  Source  Selection 
Advisory  Council  (SSAC) .  The  SSAC  reviews  the  SSEB 
recommendations  and  in-turn  submits  the  recommendations  to 
the  Source  Selection  Authority  (SSA) .  The  SSA  makes  the 
final  source  selection,  and  the  Contracting  Officer  executes 
the  selected  firm's  contract  for  the  government.  The  SSA 
sends  a  short  letter  notifying  the  unsuccessful  contractors 
of  the  contract  award.  Formal  debriefings  are  held  for 
unsuccessful  contractors  if  requested. 


The  Conventional  Approach.  The  conventional  approach 
consists  of  three  main  phases.  The  following  is  an  overview 
of  each  of  these  phases. 

Phase  One  -  Evaluation  of  Proposals .  After  the 
cut-off  date  stated  in  the  RFP,  the  SSEB  grades  each 
submitted  proposal  against  established  evaluation  criteria. 
Usually,  the  SSEB  is  broken  down  into  panels  that  correspond 
to  the  functional  areas  outlined  in  the  RFP.  The  four  main 
functional  areas  are  technical,  management,  cost,  and 
logistics  (14:198).  The  functional  panels  perform  an 
independent  evaluation  of  each  proposal  and  assign  an 
objective  rating  based  on  the  previously  defined  standards 
for  each  criteria  element.  "Evaluators  must  not  make  the 
mistake  of  rating  proposals  in  relation  to  each  other" 

(2:50)  .  Next,  the  SSEB  formulates  the  Proposal  Analysis 
Report  (PAR)  which  summarizes  the  facts  and  findings  of  how 
well  each  company  met  the  requirements  reflected  in  the  RFP 
(2:59)  .  The  PAR  is  prepared  for  the  Source  Selection 
Advisory  Council  (SSAC)  and  the  Source  Selection  Authority. 

The  SSEB  membership,  a  group  of  military  and  civilian 
personnel,  is  based  solely  on  the  functional  areas  and  the 
evaluation  criteria  defined  in  the  Source  Selection  Plan. 

The  criteria  define  "the  disciplines  that  require  analysis 
and  the  skills  within  these  disciplines"  (2:49)  .  Each 
source  selection  requires  a  unique  combination  of  personnel 
to  conduct  the  evaluation,  and  therefore,  no  permanent  board 
for  source  selection  is  maintained. 


SSEB  duties  will  take  precedence  over  the 
evaluator's  normal  duty  assignment.  All 
members  are  required  to  be  available  for 
the  entire  evaluation  period  which  will 
commence  with  the  first  meeting  of  the 
SSEB  membership  and  extend  through  the 
final  preparation  of  findings  to  the  Source 
Selection  Authority.  [2:49] 

Phase  Two  -  Fact  Findinq  and  Neqotiations 


"competitive  range"  includes  all  contractor  proposals  that 

fall  between  a  government  established  minimum  and  maximum 

requirement  level.  The  Contracting  Officer,  the  official 

point  of  contact  between  the  contractors  and  the  government, 

establishes  the  competitive  range  based  upon  the  results  of 

the  initial  SSEB  evaluation  of  the  proposals  (14:204) .  Only 

those  proposals  within  the  competitive  range  continue  into 

parallel  negotiations. 

Negotiation  involves  bargaining  between  the 
contractor  and  Contracting  Officer  to  arrive 
at  a  common  understanding  on  contract 
essentials  such  as  delivery  schedules, 
specification,  prices,  and  terms.  [14:60]. 

During  the  proposal  evaluation  phase,  the  SSEB 

generates  a  number  of  Deficiency  Reports  (DR)  and  Contractor 

Inquiries  (Cl) .  A  DR  records  a  part  of  the  proposal  that 

does  not  meet  the  minimum  requirements  of  the  RFP.  A  Cl 

seeks  information  in  addition  to  the  proposal  or  a 

clarification  of  a  part  of  the  proposal.  DRs  and  CIs  are 

submitted  through  the  Contracting  Officer  to  the  contractor 

and  must  be  satisfactorily  resolved  before  a  suitable 


contract  can  be  negotiated  (2:63)  . 


Parallel  negotiations  are  formally  ended  at  a  date 
common  to  all  contractors  and  the  results  of  the 
negotiations  are  confirmed  by  the  contractors  submitting 
their  "best  and  final  offer"  (17 : I— 1 ; 1—2 ) .  A  signed 
contract  must  be  included  with  the  offer.  Upon  receipt  of 
the  best  and  final  offers,  the  government  will  not  hold  any 
further  discussions  with  the  contractors  "unless  it  is 
clearly  in  the  Government's  interest  to  do  so  [or  the] 
information  available  at  that  time  is  inadequate  to 
reasonably  justify  contractor  selection"  (25:16940). 

Phase  Three  -  Selection  of  Contractor .  The  Source 
Selection  Authority  reviews  the  best  and  final  offers  as 
well  as  all  other  reports  pertaining  to  the  proposals  before 
selecting  the  source.  The  SSA  chooses  the  proposal  that  is 
"most  advantageous  to  the  Government  consistent  with  the 
established  evaluation  factors"  (25:16940) .  When  the 
decision  is  made,  the  winning  contractor  is  notified  and  the 
Contracting  Officer  formally  executes  the  contract  by 
signing  it.  All  unsuccessful  contractors  receive  written 
notification  of  the  contract  award.  Details  concerning  the 
rejection  of  a  contractor's  proposal  are  not  disclosed  in 
the  written  notification.  The  contractor  must  request  a 
formal  debriefing  from  the  government  to  obtain  this 
information. 
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Overview  of  the  Streamlined  Approaches 

Or igin  of  Streamlined  Approaches.  The  streamlined 
source  selection  processes  came  into  being  as  a  result  of  a 
need  to  reduce  the  time  and  manpower  requirements  associated 
with  the  use  of  the  conventional  approach.  Historically, 
the  conventional  approach  divided  the  SSEB  organization  into 
"Areas,  Items,  and  Factors  which  have  been  established  in 
the  evaluation  criteria"  (2:10).  According  to  the  handbook. 
The  Source  Selection  Process  (2)  printed  by  Aeronautical 
Systems  Division,  the  division  of  the  SSEB  organization  has 
resulted  in  large  numbers  of  personnel  gathering  to  evaluate 
contractors'  proposals  over  prolonged  periods  of  time.  The 
result  of  these  evaluations  has  been  "voluminous  reports  at 
each  level  of  the  evaluation  without  regard  to  the  probable 
impact  of  the  factor  on  the  selection  process"  (2:10)  .  in 
addition,  much  of  the  data  is  fragmented  and  overemphasizes 
"relatively  small  strengths  and  weaknesses  which  have  little 
or  no  impact  on  the  decision  process"  (2:10) . 

Criticisms  such  as  these  led  to  the  development  of 
source  selection  approaches  which  could  shorten  the  time 
cequired  for  the  conventional  approach  and  also  reduce  the 
number  of  personnel  required  to  research  and  evaluate  the 
contractors'  proposals.  The  approaches  developed  are 
modified  forms  of  the  conventional  approach  and  have  become 
known  as  streamlined  approaches. 
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The  Streamlined  Approaches .  Presently  there  are  two 
streamlined  approaches  -  the  Aeronautical  Systems  Command 
(ASD)  approach  and  the  Ballistic  Missile  Office  (BMO) 
approach. 

The  ASD  Streamlined  Approach.  The  ASD  streamlined 

approach  as  outlined  in  ASD  Pamphlet  800-7  modifies  the 

conventional  source  selection  approach  by  allowing  the  SSA 

to  establish  a  Source  Selection  Evaluation  Committee  (SSEC) 

consisting  of  a  group  of  experienced  senior  military  and 

government  civilian  personnel.  The  head  of  the  SSEC  assumes 

"the  responsibilities  and  tasks  belonging  to  the  SSAC  and 

SSEB  Chiefs"  (2:10) .  The  SSEC  Chief  makes  decisions 

concerning  the  source  selection  process  based  upon  written 

conclusions  submitted  by  an  assigned  evaluator  in  each  of 

the  facets  of  the  program. 

.  .  .  the  evaluator  in  the  SSEC  organizational 
structure  is  cnarged  with  the  total  evaluation 
of  a  facet  of  the  program  and  given  the 
responsibility  for  acquiring  expert  advice  and 
consultation  as  may  be  required.  Evaluators 
consolidate  advice  obtained  with  their  own 
findings  and  furnish  the  SSEC  Chief  a  document 
that  summarizes  the  dominant  strengths  and 
weaknesses  of  proposals  under  review  and  the 
relative  acceptability  of  each.  [2:10] 

The  expert  advice  and  consultation  mentioned  above  can 
be  sought  from  designated  experts  either  within  the 
government  or  from  the  civilian  community.  Use  of  advisors 
and  consultants,  however,  is  recommended  only  "when  a 
particular,  narrow  problem  arises  which  an  evaluator  cannot 
resolve"  (2:10) .  The  advisor  or  consultant  is  not  required 


to  submit  a  written  statement;  each  is  only  sought  for  a 
short  discussion  with  the  evaluator. 

The  end  result  of  utilizing  the  ASD  streamlined 
approach,  in  lieu  of  the  conventional  approach,  is  that  "the 
total  committee  personnel  involved,  including  advisors  and 
consultants,  should  be  in  the  area  of  25  or  30  rather  than 
in  the  hundreds"  (2:10)  as  utilized  by  conventional  source 
selections. 

The  BMP  Streamlined  Approach.  The  BMO  streamlined 

approach  modifies  the  conventional  approach  in  four  ways. 

First,  the  BMO  streamlined  approach  limits  the  number  of 

pages  a  contractor  can  submit  in  a  proposal  for  evaluation. 

In  the  past,  contractors  were  given  a  "suggested  page  count" 

for  proposals,  but  contractors  for  the  most  part  ignored 

this  suggested  limit.  Contractors  continued  to  submit 

volumes  of  information  which  cost  the  contractors  money  to 

generate  and  also  made  the  job  of  proposal  evaluation  more 

time  consuming  and  difficult.  Under  the  BMO  streamlined 

approach,  it  is  stated  emphatically  that  the  evaluators  will 

only  read  up  to  the  specified  maximum  number  of  pages. 

"Pages  in  excess  of  the  maximum  are  removed  from  the 

proposal  and  filed  to  insure  they  are  not  evaluated"  (13:6). 

The  goal  of  imposing  this  page  limit  on  the  contractor  is: 

reduce  technical  and  management  information  from 
our  previous  combined  high  of  350  pages  to  a 
maximum  of  100  pages  total.  The  aim  is  to 
eliminate  information  which  is  not  necessary  to 
the  decision  making  process.  We  do,  however, 
tailor  page  amounts  to  the  acquisitions  (e.g.. 
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on  large  dollar  value,  complicated  acquisitions 
we  may  require  150  pages  of  technical  and 
management  information) . [13 : 5] 

Secondly,  the  BMO  streamlined  approach  reduces  the 
number  of  SSEB  members  to  a  maximum  of  25  and  replaces  the 
functional  area  teams  with  a  single  evaluation  panel  (13:9)  . 
Each  member  of  the  evaluation  panel  is  required  to  be 
experienced  in  many  disciplines  related  to  the  particular 
source  selection  being  conducted.  Each  panel  member 
evaluates  all  aspects  of  a  submitted  proposal  except  cost 
considerations  which  are  evaluated  separately,  in  some 
cases  where  the  SSEB  and  SSAC  have  been  combined,  the  panel 
members  will  also  have  to  perform  comparative  analysis 
between  the  proposals. 

Thirdly,  the  BMO  streamlined  approach  establishes  a 
nine  week  time  standard  from  proposal  receipt  to  contract 
award.  The  following  is  an  example  of  how  a  source 
selection  could  be  accomplished  in  a  nine  week  period. 

TABLE  2.2 

Example  Timeline  For  BMO  Streamlined  Source  Selection 

[13:18] 


EVENTS 

TIME (WEEKS) 

FROM 

TO 

Proposal  rec 

:eipt  Competitive  range 

3 

Best  and  final  offer  4 

2 


Proposal  receipt 
Competitive  range 
Best  and  final  offer 


Contract  award 


TOTAL 


9 


Lastly,  the  BMO  streamlined  approach  adds  contractor 
presentations  to  the  source  selection  process  in  order  to 
reduce  the  total  reliance  on  written  proposals.  The 
presentations  take  place  after  the  competitive  range  is 
established.  The  purpose  of  the  presentation  \s  to  allow 
panel  members  to  gain  greater  insight  into  what  each 
contractor  is  really  offering  the  government  and  also  to 
provide  panel  members  with  a  question  and  answer  session  to 
help  resolve  proposal  deficiencies.  The  results  of  these 
discussions  are  included  in  the  final  proposal. 

Overview  of  the  Four  Step  Approach 

Four  Step  Origin.  The  four  step  source  selection 
process  was  first  proposed  by  Mr.  James  Plummer,  an  Under 
Secretary  of  the  Air  Force  (17:1-2,4).  It  was  developed  in 
response  to  criticism  from  the  Department  of  Defense  and 
industry  that  the  conventional  approach  was  too  complex, 
took  too  long  to  acquire  the  system,  and  "tended  to  obscure 
technical  and  management  differences  between  competing 
offers"  (17:1-2,4) . 

Mr.  Plummer's  four  step  approach  was  studied  by  the 
Department  of  the  Air  Force  while  used  in  seventeen  source 
selections  over  a  period  of  two  years  (17:ii).  The 
favorable  results  of  the  study  led  to  the  issuance  of  the 
September  1978  Defense  Acquisition  Circular  (DAC)  which 
authorized  the  four  step  procedures  to  be  included  in  the 
Defense  Acquisition  Regulations  (DAR)  (22:3). 
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When  the  Four  Step  is  Applicable.  The  DAR  made  the 
four  step  approach  "mandatory  for  all  competitively 
negotiated  research  and  development  acquisitions"  (2:206) 
with  some  exceptions.  Recently,  however,  the  Federal 
Acquisition  Regulation  Supplement  paragraph  15.613  deleted 
the  mandatory  use  of  the  approach  and  leaves  the  choice  of 
the  procedure  to  "the  discretion  of  the  Contracting  Officer 
for  competitively  negotiated  research  and  development 
acquisitions  with  an  estimated  value  of  two  million  dollars 
or  more"  (24:15.6-2).  The  FAR  Supplement  further  states 
that  the  four  step  approach  should  not  be  used  for 
acquisitions  where  extensive  negotiations  are  anticipated 
and  that  the  procedures  will  not  be  used  for  any 
acquisitions  whicn: 

1.  are  negotiated  pursuant  to  FAR  15.202  [Public 
exigency] ; 

2.  are  solely  for  personnel  services; 

3.  are  for  architect-engineer  services;  or 

4.  have  an  estimated  value  of  less  than 
two  million  dollars.  [24:15,6-2] 

The  Four  Step  Approach .  The  four  step  approach  to 
source  selection,  as  the  name  suggests,  consists  of  four 
main  steps.  These  steps  are  defined  in  FAR  Supplement 
paragraph  15.613  and  are  further  discussed  in  Guide  For 
Weapon  Systems  Acquisition  (14:205,206)  ,  4-Step  Source 
Selection  (17:1-10,11) ,  and  The  Source  Selection  Process 


approach  has  been  compiled  from  these  sources  and  is 
intended  to  briefly  summarize  each  step. 

Step  One  -  Evaluation  and  Discussion  of  Technical 
Proposals .  In  step  one,  technical  proposals  are  solicited 
from  all  contractors  desiring  to  bid  on  the  construction  of 
a  new  system.  The  technical  proposals  are  evaluated  on  the 
basis  of  how  well  they  satisfy  Government  requirements,  but 
no  deficiencies  in  a  proposal  are  discussed  with  the 
contractor.  The  only  discussion  allowed  is  for  the  purpose 
of  achieving  understanding  and  clarification  of  the 
contractor's  proposal.  "Cost  estimates  which  illustrate  the 
impact  of  tradeoffs  upon  projected  production  and 
operating/support  costs  are  required"  (14:205).  After  the 
evaluations  are  completed,  unqualified  bidders  are 
eliminated. 

Step  Two  -  Evaluation  and  Discussion  of  Cost/Price 
Proposals.  After  all  technical  proposals  are  analyzed,  a 
fully  documented  cost/price  proposal  is  obtained  from  the 
bidding  contractors.  Using  the  cost/price  proposals,  a 
competitive  range  is  established.  The  competitive  range  is 
based  upon  the  feasibility  of  a  system  being  procured  for  a 
given  amount  of  money.  If  a  contractor's  bid  is  outside  the 
competitive  range,  he  is  eliminated  from  the  selection 
process  and  notified  that  he  is  no  longer  in  contention  for 
the  contract.  Cost/price  discussions  are  then  held  with  all 
remaining  contractors.  The  discussions  may  serve  to  correct 


mathematical  errors,  correlate  cost  to  technical  efforts,  or 
clarify  the  Government's  requirements  for  logistic  support, 
performance  factor  trade-offs,  or  delivery  schedules 
(24:15.6.4).  Following  these  discussions  a  contractor  can 
still  be  eliminated  from  the  selection  process  for  the 
following  reasons: 

1.  when  the  proposal  was  initially  included  in 
the  competitive  range  because  it  might  have 
been  susceptible  of  being  made  acceptable;  or 

2.  because  there  was  uncertainty  whether  it 
was  in  the  competitive  range,  and  in  either 
case,  through  discussions  relating  to  ambiguities 
and  omissions,  it  becomes  clear  that  the  proposal 
should  not  have  been  included  in  the  competitive 
range  initially.  [24:15.6-4] 

Contractors  are  not  advised  of  how  high  or  low  their  bid 
fell  in  the  competitive  range  or  of  proposal  deficiencies. 

Step  Three  -  Common  Cutoff.  During  step  three,  a 
common  cutoff  date  is  established  for  the  receipt  of  all 
technical  and  cost/price  proposal  revisions.  If  a  revision 
is  made,  it  must  be  fully  documented  with  supporting  data 
describing  the  conditions  that  brought  about  the  change.  An 
unsubstantiated  revision  will  not  be  accepted.  After  the 
cutoff  date,  no  additional  proposals  or  revisions  will  be 
solicited  unless  approved  by  an  official  at  a  level  no  lower 
than  the  head  of  a  contracting  activity.  Repetitive  calls 
for  cost/price  proposals  in  order  to  auction  the  contract  to 
the  lowest  bidder  are  not  allowed. 

After  the  cutoff  date,  all  proposals  and  revisions  are 
evaluated  and  a  single  contractor  is  selected  to  enter  into 


final  negotiations.  (More  than  one  contractor  can  be 
selected  if  multiple  sources  are  desired.)  The  selection  is 
"based  on  an  integrated  decision,  involving  consideration  of 
technical  approach,  capability,  management,  design  to  cost, 
operating  and  support  cost  objectives,  historical 
performance,  price/cost  and  other  factors"  (24:15.6-5). 
Following  the  selection,  all  contractors  are  notified  of  the 
decision. 

Step  Four  -  Final  Negotiations  and  Contract  Award. 
During  the  final  negotiations,  the  Government  and  the 
selected  contractor  strive  for  a  definitive  contract 
acceptable  to  both  parties.  The  negotiations  attempt  to 
disclose  and  resolve  all  technical  deficiencies  and 
unsubstantiated  areas  of  cost.  They  do  not  involve 
"material  changes  in  the  Government's  requirements  or  the 
contractor's  proposal  which  would  affect  the  basis  for  the 
source  selection"  (24:15.613).  If  such  changes  are 
necessary,  FAR  15.606  contains  procedures  to  be  followed. 

In  the  event  the  Government  and  the  contractor  cannot  agree 
on  a  definitive  contract  before  a  mutually  established 
deadline,  the  Government  has  the  option  to  terminate  the 
negotiations  and  choose  another  contractor.  "The  final 
nagotiated  contract  must  represent  a  reasonable  probability 
that  the  Government's  requirements  will  be  satisfied  at  a 
fair  and  reasonable  cost/fee  or  price"  (24:15.6-6).  After 
the  contract  is  signed,  all  eliminated  contractors  desiring 
to  know  their  proposal  deficiencies  are  formally  debriefed. 


Ill .  Methodology 


Introduction 

The  research  project  was  conducted  in  three  separate 
phases.  The  first  phase  involved  telephone  and  personal 
interviews  to  determine  if  a  sufficient  number  of  source 
selection  records  and  experienced  source  selection  personnel 
would  be  available  to  permit  the  research.  The  second  phase 
was  a  review  of  completed  source  selection  records  at 
different  AFSC  product  divisions.  In  the  third  phase,  a 
mail  survey  was  used  to  gather  background  and  opinion  data 
on  different  aspects  of  the  source  selection  process.  The 
surveys  were  sent  to  personnel  in  the  AFSC  product  divisions 
experienced  in  the  source  selection  process.  Each  phase  is 
described  in  detail  below. 

Phase  I  -  Data  Availability 

In  order  to  accomplish  research  objective  one,  we  first 
had  to  determine  if  the  required  source  selection  records 
and  names  of  experienced  source  selection  personnel  were 
available.  Telephone  and  personal  interviews  with  the 
Source  Selection  Department  Chiefs  at  the  Aeronautical 
Systems  Division  (ASD) ,  Electronic  Systems  Division  (ESD)  , 
Space  Division  (SD) ,  and  Eastern  Space  and  Missile  Center 
(ESMC)  were  conducted.  In  the  interviews,  we  were  looking 
for  two  general  areas  of  information.  First,  we  had  to 


determine  the  accessibility  of  the  source  selection  records 


required  to  meet  research  objective  two.  The  initial 
records  review  research  focused  on  the  four  step  approach 
because  it  has  seldom  been  used  by  the  Air  Force  since  the 
approach’s  release  in  1978.  The  records  had  to  contain  the 
following  information  to  accomplish  this  objective: 

a)  the  number  of  completed  four  step  source  selections  in 
each  product  division  since  1978; 

b)  the  names  of  the  principal  points-of-contact  for 
completed  four  step  source  selections;  and 

c)  the  names  of  any  other  personnel  who  might  provide 
insight,  comments,  or  opinions  on  the  four  step  source 
selection  process. 

The  second  general  area  of  information  required  was  to 
assess  the  difficulty  involved  with  generating  a  mailing 
list  of  experienced  source  selection  personnel.  This 
information  was  critical  if  the  mail  survey  questionnaire 
phase  of  the  research  was  to  be  accomplished. 

Finally,  after  reviewing  the  results  of  the  source 
selection  personnel  interviews,  a  decision  had  to  be  made  o 
whether  the  research  objectives  could  be  accomplished. 

Phase  II  -  Records  Review 

Data  Collection.  A  review  of  completed  source 
selection  records  of  both  the  conventional  and  four  step 
approaches  was  conducted  at  the  different  AFSC  product 
divisions.  The  data  gathered  during  the  records  review  was 
required  in  order  to  accomplish  research  objective  two. 
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Due  to  the  limited  number  of  four  step  source 
selections  since  1978/  we  decided  to  attempt  a  review  of  all 
four  step  records.  Only  a  random  sampling  of  the 
conventional  approach  records  would  be  evaluated  due  to  the 
large  number  of  conventional  source  selections  occurring 
each  year. 

The  following  list  contains  the  data  required  from  each 
source  selection  record: 

a)  the  number  of  personnel  serving  on  the  Source  Selection 
Advisory  Council  (SSAC)  and  Source  Selection  Evaluation 
Board  (SSEB) ; 

b)  the  level  of  the  Source  Selection  Authority  (SSA) ; 

c)  the  security  classification  of  proposals; 

d)  the  final  contract  award  price; 

e)  the  total  time  (in  days)  from  RFP  release  to  final 
contract  award  announcement; 

f)  the  product  division  responsible  for  the  source 
selection; 

g)  the  type. of  acquisition  -  system,  subsystem,  or 
modification; 

h)  the  number  of  bidders  submitting  proposals  to  the  SSEB; 

i)  the  average  number  of  pages  in  each  submitted  proposal 
(broken  down  into  functional  areas  if  possible  -  technical, 
cost,  management,  and  logistics  support) ; 

j)  the  source  selection  approach  used;  and 


k)  the  contract  type  -  Fixed-Price  plus  Incentive,  Cost 
plus  Incentive,  Fixed-Price. 

Using  the  Data  to  Meet  Research  Objective  2.  Research 
objective  two  attempted  to  match  a  completed  four  step 
acquisition  to  an  equivalent  conventional  acquisition.  After 
the  match  was  made,  contract  relationships  would  be  used  to 
characterize  a  specific  approach's  use.  The  following  list 
is  the  criteria  used  for  matching  similar  four  step  and 
conventional  source  selections: 

1)  the  product  division  responsible; 

2)  the  type  of  contract; 

3)  the  type  of  acquisition;  and 

4)  the  final  contract  award  price. 

Once  the  matching  of  approaches  was  accomplished,  an 
analysis  would  draw  conclusions  based  on  the  following 
areas: 

a)  number  of  bidders; 

b)  average  number  of  pages  in  each  submitted  proposal; 

c)  number  of  people  on  the  SSEB;  and 

d)  total  time  from  RFP  release  to  final  contract. 

Phase  III  -  Mail  Survey  Questionnaire 

Data  Collection  and  Analysis .  Two  sets  of  data  were 
collected  during  this  phase  of  the  research.  First,  the 
mailing  list  for  the  survey  was  generated.  Second,  the 
survey  questionnaire  was  constructed,  sent  to  the  field,  and 
the  survey  responses  analyzed. 


Mailing  List  Construction.  Due  to  the  time 
limitations  on  the  research  team  and  the  transitory  nature 
of  the  AFSC  job  environment,  it  was  impossible  to  identify 
the  entire  source  selection  population  of  interest. 
Therefore,  a  mailing  list  (sample)  representative  of  the 
source  selection  population  was  compiled.  The  sample  had  to 
consist  of  people  from  different  AFSC  product  divisions, 
diverse  functional  areas,  and  different  military/civilian 
ranks.  A  nonrandom  (nonprobabilistic)  sample  selection  plan 
was  used  for  two  reasons.  First,  people  experienced  in  the 
source  selection  process  were  in  the  best  position  to 
provide  the  information  needed.  According  to  Emory,  the 
requirement  for  expert  judgment  or  opinion  often 
necessitates  the  use  of  nonrandom  sample  selection  (18:88). 
The  second  reason  was  that  the  entire  population  could  not 
be  identified,  thus  eliminating  the  possibility  of  any  type 
of  random  sampling. 

The  researchers  decided  on  a  two-step  process  for 
generating  the  mailing  list.  The  first  step  was  to  ask  each 
Source  Selection  Department  Chief  to  provide  the  names  and 
phone  numbers  of  20  experienced  source  selection  personnel 
from  their  respective  product  divisions.  Letters  with  the 
request  were  sent  to  the  following  offices: 

1.  Aeronautical  Systems  Division  (ASD/PMP) 

2.  Electronic  Systems  Division  (ESD/PKP-2) 

3.  Space  Division  (SD/PMOM) 
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4.  Armament  Division  (AD/PMP) 

5.  Ballistic  Missile  Office  (BMO/PMP) 

6.  Eastern  Space  and  Missile  Center  (ESMC/PMP) 

The  team  recognized  that  using  this  method  of  sampling 
presented  the  possibility  of  a  bias  towards  a  particular 
approach.  In  an  attempt  to  eliminate  this  bias,  the  letter 
to  the  Department  Chiefs  did  not  disclose  any  details  of  the 
survey  subject  matter. 

The  second  step  in  the  survey  mailing  process  was  to 
expand  the  name  list  to  200  names.  This  step  involved 
making  telephone  calls  to  the  people  identified  by  the 
Department  Chiefs  and  requesting  2  or  3  additional  names  of 
experienced  people  in  the  source  selection  process.  The 
telephone  calls  served  two  purposes.  First,  they  allowed  us 
to  expand  our  mailing  list  and  second,  they  allowed  us  to 
verify  the  experience  of  each  source  selection  personnel 
identified  by  the  Department  Chiefs.  Again,  care  was  taken 
not  to  divulge  the  subject  matter  of  the  survey  in  an 
attempt  to  keep  approach  bias  to  a  minimum. 

Survey  Construction,  Mailing ,  and  Analysis.  This 
section  of  Phase  III  methodology  used  a  mail  survey  and  the 
Statistical  Package  for  the  Social  Sciences  (SPSS)  computer 
package.  The  section  consists  of  3  steps.  The  first  step 
involved  preparing  the  survey  instrument  and  obtaining  Air 
Force  approval  to  administer  the  survey.  The  second  step 
consisted  of  mailing  the  survey  to  the  208  source  selection 
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personnel  identified  in  the  previous  data  collection 
section.  Finally,  the  results  of  the  survey  were  analyzed 
and  conclusions  drawn  from  the  analysis.  The  following  is  a 
discussion  of  the  three  steps. 

Survey  Instrument  Preparation.  A  mail  survey 
questionnaire  was  chosen  as  the  primary  instrument  to  gather 
source  selection  data  not  currently  available  from 
experienced  source  selection  personnel.  Two  possible 
methods  of  data  collection  were  considered,  the  mail  survey 
and  the  telephone  interview.  Telephone  interviews  have  an 
excellent  response  rate,  but  can  be  very  time-consuming 
(a:39) .  A  trial  run  of  a  typical  telephone  interview 
required  45  minutes  or  more  of  the  interviewee's  time.  A 
mail  survey  covering  the  same  questions  would  only  require 
15  minutes.  Since  indepth  comments  were  not  required  by  the 
team,  the  mail  survey  was  chosen  as  the  least  costly  method 
in  terms  of  manpower  and  time. 

Next,  the  actual  questions  were  formulated.  Respondent 
background  characteristics  and  expert  opinions/preferences 
were  the  two  types  of  information  required  to  meet  our 
research  objectives.  Background  information  (demographics) 
are  contained  in  Part  I,  questions  A  thru  L  of  the  survey. 
The  background  information  was  required  for  categorizing  the 
survey  results  during  the  analysis  step.  The  questions  in 
Part  II  of  the  survey  were  the  result  of  the  research  team's 
perceptions  of  the  problems  and  issues  facing  source 


were  formulated  after  extensive  interviews  with  the  source 
selection  department  chiefs  and  other  experienced  source 
selection  personnel  in  AFSC.  Part  II  dealt  with  four 
general  areas  of  opinions  and  preferences.  The  first  area 
questioned  the  participant’s  preferred  source  selection 
approach  when  associated  with  a  specific  system 
characteristic  (Part  II,  questions  A,  B1  thru  B7) .  The 
second  question  area  dealt  with  the  time  and  manpower 
requirements  of  the  various  approaches  (Part  II,  questions 
Cl  thru  C4) .  The  third  question  area  addressed  the  problem 
of  using  the  source  selection  process  to  further  define 
system  requirements  (Part  II,  questions  D  and  H) .  Source 
selection  personnel  perceptions  of  the  current  attempts  to 
improve  the  source  selection  process  were  covered  in  the 
fourth  area  of  the  survey  (Part  II,  questions  E,F,G,I,J,  and 
K) .  The  final  section  titled  "Other  Comments"  invited 
respondents  to  express  additional  opinions  on  the  source 
selection  process. 

After  preparing  the  survey,  the  next  step  was  to  test 
the  survey  instrument.  Two  intermediate  Program  Management 
(SYS  400)  Professional  Continuing  Education  courses  at  AFIT 
were  selected  for  test  runs.  The  majority  of  personnel 
attending  this  course  were  field  grade  officers  or  civilian 
equivalents  with  acquisition  experience.  Based  on  the 
comments  received  from  the  classes,  changes  were  made  as 
necessary  to  improve  the  survey  instrument. 


The  final  step  was  to  obtain  approval  from  the  Air 
Force  Survey  Control  office  to  administer  it.  A  survey 
package  was  prepared  and  submitted  for  approval.  The 
package  included  a  description  of  the  purpose  and  objectives 
of  the  survey,  a  description  of  the  population  and  sample, 
an  estimate  of  the  total  costs  to  be  incurred,  a  copy  of  the 
survey  cover  letter,  and  a  copy  of  the  survey. 

Administering  the  Survey.  Once  the  Air  Force 
Survey  Control  Office  approved  the  survey,  the  team  prepared 
a  survey  packet  for  each  survey  participant.  The  survey 
packet  contained  a  copy  of  the  cover  letter,  a  survey,  and  a 
return  envelope.  The  cover  letter  requested  that  the 
surveys  be  returned  within  7  days  of  receipt.  Appendix  A 
contains  a  copy  of  the  cover  letter  sent  to  ASD  personnel 
and  appendix  B  contains  a  copy  of  the  mail  survey 
questionnaire.  A  special  cover  letter  was  prepared  for  each 
product  division  in  order  to  show  the  team's  coordination 
with  each  product  division  source  selection  Department 
Chief . 

Analysis  of  Results .  This  section  presents  a 
brief  overview  of  the  statistical  techniques  used  during  the 
analysis  of  the  survey  responses.  The  actual  analysis 
techniques  applied  to  each  research  objective  will  be 
discussed  later  in  this  chapter. 

The  SPSS  computer  package  on  the  ASD  Cyber  computer 
system  was  used  to  store  the  coded  survey  responses  and 
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generate  the  required  statistics.  Returned  surveys  were 
numbered  sequentially  and  dated  upon  receipt.  The  responses 
to  the  questions  were  coded  and  entered  into  a  Cyber  data 
file.  The  date  the  survey  was  input  into  the  data  file  was 
also  placed  on  the  survey. 

The  survey  responses,  as  coded  into  the  computer  data 
file,  were  examined  for  their  level  of  measurement.  The 
level  of  measurement  determines  which  statistical  techniques 
can  be  applied  to  the  data.  According  to  S.S.  Steven,  there 
are  four  traditional  levels  of  measurement:  nominal, 
ordinal,  interval,  and  ratio.  The  level  of  measurement 
assigned  to  a  data  field  is  a  result  of  the  ordering  and 
distance  properties  of  the  data.  All  data  fields  associated 
with  the  survey  responses  were  limited  to  the  nominal 
(lowest)  level  of  measurement,  except  the  Yes/No  questions 
which  are  discussed  later  in  Chapter  III.  In  the  nominal 
level  of  measurement,  the  values  divide  the  data  field  into 
distinct  categories  and  each  data  field  "value  itself  serves 
merely  as  a  label  or  name  for  the  category".  (15:4) 

Oue  to  the  nominal  level  of  measurement  of  the  data 
fields,  two  SPSS  routines  were  selected  for  use: 

FREQUENCIES  and  CROSSTABS.  FREQUENCIES  calculate  one-way 
frequency  distribution  tables  on  the  data  fields.  The 
tables  are  used  to  give  the  researcher  a  general  idea  of  how 
the  responses  are  distributed.  An  understanding  of  the 
distribution  characteristics  of  a  data  field  is  important 


before  progressing  to  the  analysis  of  the  relationships 
between  data  fields  (15:194).  SPSS  was  used  to  generate  a 
simple  frequency  distribution  table  and  a  histogram  for  each 
data  field.  Additional  statistics,  such  as  the  measures  of 
central  tendencies,  were  not  generated  because  they  are 
meaningless  for  nominal  (categorical)  data. 

After  analyzing  the  frequency  distribution  tables, 
relationships  between  the  background  information  and  opinion 
responses  were  established  to  analyze  the  data  in  terras  of 
the  research  objectives.  One  statistical  technique 
available  for  the  analysis  of  nominal  data  field 
relationships  is  CROSSTABS.  Using  CROSSTABS  produces  a 
contingency  table  that  is  a  "joint  frequency  distribution  of 
cases  according  to  two  or  more  classif icatory  variables" 
(15:218) . 

The  contingency  table  is  an  extremely  versatile  and 
valuable  instrument  which  paradoxically  finds  frequent 
inappropriate  employment  because  of  its  simplicity  and 
yet  is  often  erroneously  overlooked  in  favor  of 
complex  parametric  models  requiring  highly 
questionable  assumptions  about  the  data.  (27:196) 

A  contingency  table  provides  a  row  by  column 
representation  of  the  frequency  of  occurrence  between 
different  categories  of  two  data  fields.  For  example,  a 
contingency  table  of  product  division  by  preferred  approach 
would  have  the  different  product  divisions  listed  above  the 
columns  and  the  different  approaches  listed  down  the  side  of 
the  table.  The  intersection  of  a  row  and  a  column  contains 
the  percentage  of  the  product  division  personnel  who  picked 
the  approach  associated  with  that  column. 


CROSSTABS  was  used  to  examine  the  relationships  between 
the  background  characteristics  of  the  respondents  and  the 
opinion/preference  questions  that  involved  a  selection  of  a 
particular  approach. 

The  Yes/No  type  questions  allowed  the  team  more 

latitude  in  the  selection  of  statistical  techniques.  The 

Yes/No  data  fields  are  dichotomies  (two  possible  values) . 

"Any  dichotomy  can  be  treated  as  though  it  were  an  interval- 

level  measure"  (15:5)  .  Therefore,  we  performed  an  analysis 

of  variance  on  these  data  fields.  The  SPSS  manual  describes 

Analysis  of  Variance  as  follows: 

Analysis  of  variance  is  a  statistical  technique  that 
assesses  the  effects  of  one  or  more  categorical 
independent  variables,  measured  at  any  level  upon  a 
continuous  dependent  variable  that  is  usually  assumed 
to  be  measured  at  a  interval  level.  Conceptually,  the 
cases  are  divided  into  categories  based  on  their  values 
for  each  of  the  independent  variables,  and  the 
differences  between  the  means  of  these  cetegories  on 
the  dependent  variable  are  tested  for  statistical 
significance.  (15:9) 

For  the  purpose  of  this  research,  the  SPSS  statistical 
routine  ONEWAY  was  used  for  the  analysis  of  variance. 

ONEWAY  is  a  special  form  of  analysis  of  variance  limited  to 
only  one  independent  variable.  The  independent  variable  can 
be  nominal  but  the  dependent  variable  must  be  measured  at 
least  on  an  interval  level  (dichotomy).  Thus,  the 
background  information  data  fields  were  used  as  the 
independent  variables.  ONEWAY  shows  any  statistical 
differences  in  the  Yes/No  responses  based  on  the  background 
categories  (groupings) . 


IV.  Findings 


Introduction 

This  chapter  presents  the  results  of  the  three  phases 
of  the  research  project  described  in  Chapter  III.  The  first 
phase  involved  a  data  availability  check.  This  check 
determined  if  the  source  selection  records  needed  to 
accomplish  the  records  review  in  Phase  II  were  accessible  by 
the  team.  The  check  was  also  used  to  assess  the  difficulty 
in  generating  a  list  of  experienced  source  selection 
personnel  in  support  of  the  third  phase,  the  mail  survey 
questionnaire. 

Phase  I  -  Data  Availability 

The  primary  points-of-contact  during  Phase  I  were  the 
following  Source  Selection  Department  Chiefs:  Mr.  Jim 
Helmig,  Aeronautical  Systems  Division;  Ms.  Irene  Biddy, 
Electronic  Systems  Division;  Mr.  Joe  Kruger,  Space  Division; 
and  Mr.  Herb  Wasserman,  Eastern  Space  and  Missile  Center. 
These  Department  Chiefs  were  contacted  by  telephone  or 
visited  between  September  and  December  1984.  From  the 
contacts,  the  research  team  made  two  determinations.  First, 
an  insufficient  number  of  four  step  source  selection  records 
were  available  to  perform  Phase  II  of  the  research.  Second, 
numerous  source  selection  personnel  could  be  identified  for 
a  mail  survey  questionnaire. 


The  actual  number  of  four  step  source  selections  that 
have  occurred  since  1973  is  small.  According  to  Mr.  Helmig, 
ASD  has  initiated  only  five  four  step  source  selections.  Of 
the  five  four  steps  initiated,  the  team  could  obtain  access 
to  only  1  set  of  records.  Two  sets  of  records  were  stored 
off-base  in  a  vault,  one  set  was  classified,  and  one  four 
step  converted  to  the  conventional  approach  during  the 
source  selection  process,  in  ESD,  Ms.  Biddy  was  aware  of 
only  one  four  step  source  selection  and  it  was  currently  in¬ 
progress.  The  records  of  that  source  selection  would  not  be 
accessible  until  the  source  selection  was  complete.  No 
completion  data  was  available. 

Mr.  Herb  Wasserman  at  ESMC  was  able  to  obtain  the 
records  for  one  completed  four  step  source  selection.  A 
review  of  the  existing  four  step  source  selection  records  at 
Space  Division  revealed  only  partial  data  was  available.  In 
the  final  analysis,  the  team  decided  that  the  three  full 
sets  of  four  step  records  and  the  partial  sets  would  not  be 
sufficient  to  perform  Phase  II  of  the  research  project. 

All  the  department  chiefs  indicated  that  their  product 
divisions  conducted  numerous  source  selections  during  any 
given  year  and  that  many  people  were  involved  with  each 
source  selection.  As  a  result,  we  concluded  that  a 
sufficient  pool  of  experienced  source  selection  personnel 
existed  and  was  available  for  the  survey  questionnaire  in 


Phase  III. 


Phase  II  -  Records  Review 

During  Phase  I  of  the  research,  the  team  determined  the 
records  review  phase  could  not  be  accomplished  due  to  the 
relatively  small  number  of  completed  four  step  source 
selections,  the  incompleteness  of  the  some  records,  and  the 
inaccessibility  of  other  records. 

Phase  III  -  Mail  Survey  Questionnaire 

Phase  III  Overview.  Five  steps  were  followed  in  this 
phase  of  the  research.  They  were: 

1.  the  mailing  list  was  constructed; 

2.  the  survey  was  constructed; 

3.  Air  Force  approval  to  administer  the  survey  was 
obtained; 

4.  the  survey  was  sent  to  the  field;  and 

5.  the  responses  from  the  survey  participants  were 
analyzed . 

Step  One  -  Mail  List  Construction.  All  the  source 
selection  department  chiefs  replied  to  our  request  for  a 
list  of  personnel  experienced  in  the  source  selection 
process.  The  team  expanded  the  mailing  list  by  calling  the 
people  provided  by  the  department  chiefs  and  requesting 
additional  names.  The  completed  mail  list  contained  208 
people. 

Step  Two  -  Survey  Construction.  The  survey  was 


initially  constructed  in  September  1984.  The  survey 
instrument  was  administered  to  two  SYS  400  Professional 


Continuing  Education  (PCE)  courses  at  AFIT.  As  a  result  of 
the  comments  received  from  the  SYS  400  classes,  several 
minor  changes  to  the  instrument  were  made. 

Step  Three  -  Survey  Approval .  The  Air  Force  survey 
approval  process  took  approximately  7  weeks  due  to  the  time 
required  to  coordinate  with  local  labor  unions  at  each 
product  division.  On  May  31,  1985,  verbal  approval  was 
received  from  the  Air  Force  Survey  Control  Office  to 
administer  the  survey.  The  survey  control  number  35-41  was 
assigned  to  the  survey  and  appeared  on  each  survey  cover 
letter . 

Step  Four  -  Survey  Administration.  The  team  finalized 
the  survey  packets  and  mailed  the  surveys  on  the  same  day 
approval  was  received.  The  cut-off  date  for  acceptance  of 
surveys  was  15  July  1985. 

Step  Five  -  Survey  Results.  The  results  of  the  survey 
were  divided  into  four  main  sections.  The  first  section 
presents  the  response  rate  and  background  of  the  survey 
participants.  The  second  section  presents  the  survey 
participants  preferences  for  a  source  selection  approach  in 
different  acquisition  situations.  The  third  section 
presents  the  survey  participants'  opinion  of  the  time  and 
manpower  requirements  needed  for  different  source  selection 
approaches.  The  final  section  presents  the  survey 
participants  opinions  about  the  source  selection  process  and 
the  differing  characteristics  of  each  source  selection 
approach. 


The  descriptive  statistics  in  this  chapter  are 
presented  in  tabular  form  and  the  responses  from  several 
survey  questions  may  be  included  in  the  same  table.  All 


percentage  figures  in  the  tables  are  computed  based  upon  the 
number  o£  survey  participants  who  answered  a  specific 
question  and  do  not  reflect  the  total  number  of  survey 
participants  who  were  eligible  to  answer. 

Section  One  -  Response  Rate  and  Background.  This 
section  contains  a  summary  of  the  responses  to  the 
background  questions  from  the  survey  (Part  I) . 

Response  Rate .  The  source  selection  survey 
questionnaire  was  sent  to  208  source  selection  personnel  in 
Air  Force  Systems  Command.  One  hundred  forty  of  the 
surveyed  personnel  returned  a  completed  survey.  The  overall 
response  rate  was  67.3  percent.  Table  4.1  summarizes  the 
response  rate  by  product  division,  and  table  4.2  contains 
the  response  rate  by  military  rank  and  civilian  grade.  A 
majority  of  the  responses  were  from  the  middle  and  upper 
management  areas,  giving  the  researchers  confidence  in  the 
experience  level  of  the  population  sample. 

Background  Information.  Tables  4.3  thru  4.5 
summarize  the  demographic  questions  from  the  survey 
(questions  D,  F,  and  G) .  The  tables  show  the  highest 
education  level  achieved  by  the  survey  participants,  their 
current  functional  area,  and  the  type  of  procurement  they 


TABLE  4.1 


Product  Division  Response  Rate 


PRODUCT  SURVEYS  COMPLETED  RESPONSE 

DIVISION  MAILED  SURVEYS  RATE ( % ) 


ASD 

88 

51 

58.09 

SD 

41 

21 

51.2 

ESD 

38 

34 

89.5 

AD 

18 

14 

77.8 

BMO 

12 

7 

58.3 

OTHER 

14 

13 

92.9 

TOTAL 

208 

140 

67.3% 

TABLE  4.2 

Military  Rank  or  Civilian  Grade  of  Survey  Participants 


RANK/GRADE 

NUMBER 

PERCENT  OF  TOTAL 

COL 

6 

4.3 

LTC 

13 

9.3 

MAJ 

13 

9.3 

CAPT 

11 

7.9 

1LT 

4 

2.9 

2LT 

3 

2.1 

SES 

4 

2.9 

GM-15 

17 

12.1 

GM-14 

16 

11.4 

GM-13 

20 

14.3 

GS-14 

2 

1.4 

GS-13 

16 

11.4 

GS-12 

15 

10.7 

TOTAL 


140 


100% 


TABLE  4.3 


Education  Level  of  Survey  Participants 


DEGREE 

HUMBER 

PERCENT  OF  TOTAL 

HIGH  SCHOOL 

7 

5.0 

ASSOCIATE 

4 

2.9 

BACHELORS 

35 

25.0 

MASTERS 

85 

60.7 

DOCTORATE 

9 

6.4 

TOTAL 

140 

100% 

TABLE  4.4 

Current  Functional  Work  Area  of  Survey  Participants 


FUNCTIONAL  AREA 

NUMBER 

PERCENT  OF  TOTAL 

PROGRAM  MANAGEMENT 

37 

26.4 

CONTRACT  OR 

MANUFACTURING  MGMT 

32 

22.9 

ENGINEERING 

20 

14.3 

CONFIGURATION  MGMT 

6 

4.3 

PROGRAM  CONTROL  MGMT 

12 

8.6 

LOGISTICS  MGMT 

10 

7.1 

TEST  AND  EVALUATION 

4 

2.9 

OTHER 

19 

13.6 

TOTAL 


140 


100% 


are  most  familiar  with.  Table  4.6  shows  the  grouping  of  the 
survey  participants  by  the  total  number  of  source  selections 
they  have  participated  in.  Approximately  half  of  the 
responses  were  from  personnel  with  more  than  5  source 
selections,  again,  indicating  a  high  experience  level. 


TABLE  4.5 


Type  of  Procurement  Survey  Participants  are  Familiar  With 


PROCUREMENT  TYPE 

NUMBER 

PERCENT  OF  TOTAL 

AIRCRAFT 

25 

18.8 

ELECTRONICS 

52 

39.1 

ARMAMENT 

9 

6.8 

SPACE/MISSILES 

29 

21 .8 

OTHER 

18 

13.5 

TOTAL 

133 

100% 

TABLE  4.5 

Number  of  Source 

Selections  Survey 

Participants  Have 

Participated  In 

NUMBER  OF 

SOURCE  SELECTIONS 

NUMBER 

PERCENT  OF  TOTAL 

LESS  THAN  5 

73 

52.1 

6  TO  10 

26 

18.6 

11  TO  20 

23 

16.4 

GREATER  THAN  21 

18 

12.9 

TOTAL 

140 

100% 

Source  Selection  Experience  and  Training . 

The  results  of  survey  question  J  are  contained  in  Table  4.7 
The  table  shows  how  the  survey  participants  gain  most  of 
their  source  selection  experience.  Table  4.8  shows  the 
approaches  the  participants  are  familiar  with  (survey 
question  L) ,  and  Tables  4.9  and  4.10  summarize  the  two 


survey  questions  on  training  (H  and  I) .  A  majority  of  the 
survey  participants  did  not  receive  any  training  prior  to 
their  first  source  selection  (Table  4.10)  and  35.7%  have 
received  no  formal  Air  Force  training  in  source  selection. 
Table  4.11  shows  that  a  majority  of  the  survey  participants 
have  participated  in  a  conventional  source  selection  but 
participation  in  other  approaches  is  limited. 

TABLE  4.7 

Major  Source  of  Source  Selection  Experience 


SOURCE 

NUMBER 

PERCENT  OF  TOTAL 

FORMAL  MILITARY 
TRAINING  COURSES 

1 

.7 

ON-THE-JOB  TRAINING 

99 

71.6 

INDIVIDUAL  STUDY 

34 

24.3 

NO  EXPERIENCE 

2 

1.3 

OTHER  SOURCES 

3 

2.1 

TOTAL 

139 

100% 

TABLE  4.8 

Type  of  Source  Selection  Approaches  Survey  Participants  Are 

Familiar  With 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL  * 

CONVENTIONAL 

133 

95.0 

FOUR  STEP 

103 

73.6 

ASD  STREAMLINED 

51 

36.4 

BMO  STREAMLINED 

37 

26.4 

OTHER 

33 

23.6 

*  NOTE:  Multiple  answers  were  possible. 


TABLE  4.9 


Military  Training  Courses  Survey  Participants  Have  Attended 


TRAINING  COURSE 

NUMBER 

PERCENT  OF  TOTAL  * 

INTRO  TO  SYSTEMS  ACQUISITION 

(SYS  100) 

41 

29.3 

ACQUISITION  PLANNING  AND 

ANALYSIS  (SYS  200) 

23 

16.4 

CONTRACT  ADMINISTRATION 

(PPM  152) 

18 

12.6 

GOVERNMENT  CONTRACT 

LAW  (PPM  302) 

28 

20.0 

LOGISTICS  MGMT  (LOG  224) 

8 

5.7 

OTHER 

37 

26.4 

NO  TRAINING 

50 

35.7 

*  NOTE:  Multiple  answers 

were  possible. 

TABLE  4.10 

Did  the  Survey  Participants  Have  Source  Selection  Training 
Prior  to  their  First  Source  Selection? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

YES 

15 

11.0 

NO 

121 

89.0 

TOTAL 

136 

100% 

SSP  Generation.  Table  4.12  shows  that  a 
majority  of  the  survey  participants  have  participated  in  a 
Source  Selection  Plan  generation  (survey  question  K) .  This 
experience  level  was  important  in  the  analysis  of  research 
objective  3  -  approaches  considered  during  SSP  generation. 
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TABLE  4.11 


What  Source  Selection  Approaches  Have  the  Survey 
Participants  Participated  In? 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL  * 

CONVENTIONAL 

115 

82.1 

FOUR  STEP 

40 

28.6 

ASD  STREAMLINED 

35 

25.0 

BMO  STREAMLINED 

11 

7.9 

OTHER 

30 

21.4 

*  NOTE:  Multiple 

answers  were 

possible . 

TABLE  4.12 

Have  Survey  Participants  Participated  in 

Plan  Generation? 

a  Source  Selection 

RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

YES 

117 

83.6 

NO 

23 

16.4 

TOTAL 

140 

100% 

Section  Two  -  Preferred  Approaches .  The  following 
sections  present  the  computed  results  from  Part  II  of  the 
survey.  This  section  shows  the  participant's  preferences 
for  a  specific  source  selection  approach  under  varying 
acquisition  situations.  Table  4.13  summarizes  the  results 
of  question  A  and  shows  that,  overall,  most  survey 
participants  prefer  the  conventional  approach. 


TABLE  4.13 


Overall  Preferred  Approach 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

54 

50.9 

FOUR  STEP 

5 

4.7 

ASD  STREAMLINED 

16 

15.1 

BMO  STREAMLINED 

14 

13.2 

OTHER 

11 

10.4 

NO  PREFERENCE 

6 

5.7 

TOTAL 

106 

100% 

Preferred  Approach  -  Technology .  Tables  4.14 
and  4.15  show  the  survey  participant's  approach  preference 
given  the  two  extremes  of  technology  -  state-of-the-art 
technology  and  proven  technology  (survey  questions  B.l  and 
3.2) .  For  state-of-the-art  technology,  a  majority  of  the 
survey  participants  chose  the  conventional  approach.  For 
proven  technology,  the  survey  participants  were  evenly 
divided  between  the  conventional  approach  and  the 
streamlined  approaches. 

Preferred  Approach  -  Contract  Cost.  Tables 
4.16  thru  4.18  show  the  preferred  approaches  considering 
contract  cost  (survey  questions  B.3  to  B.5).  The  contract 
cost  options  were  less  than  $1  million,  between  $1  and  $5 
million,  and  more  than  $5  million.  Only  in  the  case  of  a 
contract  cost  of  more  than  $5  million  was  an  approach 
selected  by  a  majority  of  the  survey  participants.  In  this 
case,  the  conventional  approach  was  preferred. 


TABLE  4.14 


Preferred  Approach  for  State-of-the-Ar t  Technology 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

55 

51.4 

FOUR  STEP 

14 

13.1 

ASD  STREAMLINED 

11 

10.3 

BMO  STREAMLINED 

14 

13.1 

OTHER 

4 

3.7 

NO  PREFERENCE 

9 

8.4 

TOTAL 

107 

100% 

TABLE  4.15 

Preferred  Approach  for  Proven 

Technology 

APPROACH  NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

39 

36.8 

FOUR  STEP 

10 

9.4 

ASD  STREAMLINED 

21 

19.8 

BMO  STREAMLINED 

17 

16.0 

OTHER 

8 

7.5 

NO  PREFERENCE 

11 

10.4 

TOTAL 

106 

100% 

Preferred  Approach  -  Number  of  Bidders . 
Tables  4.19  and  4.20  summarize  the  preferred  approach  based 
on  the  number  of  expected  bidders  in  response  to  the  RFP 
(survey  questions  B.6  and  B.7) .  The  number  of  bidders  did 
not  appear  to  effect  approach  choice. 


APPROACH 


NUMBER 


PERCENT  OF  TOTAL 


CONVENTIONAL 

19 

17.9 

FOUR  STEP 

9 

8.5 

ASD  STREAMLINED 

28 

26.4 

BMO  STREAMLINED 

21 

19.8 

OTHER 

18 

17.0 

NO  PREFERENCE 

11 

10.4 

TOTAL 

106 

100% 

TABLE  4.17 

Preferred  Approach  for  a  Contract  between  1  and  5  Million 

Dollars 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

30 

28.3 

FOUR  STEP 

7 

6.6 

ASD  STREAMLINED 

24 

22.6 

BMO  STREAMLINED 

19 

17.9 

OTHER 

16 

15.1 

NO  PREFERENCE 

10 

9.4 

TOTAL 


106 


100% 


APPROACH 


NUMBER 


PERCENT  OF  TOTAL 


CONVENTIONAL 

59 

55.7 

FOUR  STEP 

5 

4.7 

ASD  STREAMLINED 

11 

10.4 

BMO  STREAMLINED 

13 

12.3 

OTHER 

14 

13.2 

NO  PREFERENCE 

4 

3.8 

TOTAL 

106 

100% 

TABLE  4.19 

Preferred  Approach  for  Four  or  more  Bidders 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

39 

37.1 

FOUR  STEP 

15 

14.3 

ASD  STREAMLINED 

17 

16.2 

BMO  STREAMLINED 

17 

16.2 

OTHER 

5 

4.8 

NO  PREFERENCE 

12 

11.4 

TOTAL 


105 


100% 


TABLE  4.20 


Preferred  Approach  for  less  than  Four  Bidders 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

48 

45.7 

FOUR  STEP 

7 

6.7 

ASD  STREAMLINED 

16 

15.2 

BMO  STREAMLINED 

16 

15.2 

OTHER 

6 

5.7 

NO  PREFERENCE 

12 

11.4 

TOTAL 

105 

100% 

Section  Three  -  Time  and  Manpower  Requirements. 
Section  three  presents  the  computed  results  of  the  survey 
participants  opinions  of  the  time  and  manpower  requirements 
each  approach  requires.  Table  4.21  summarizes  the  results 
of  survey  question  C.l  (longest  time  between  proposal 
receipt  and  contract  award) ,  Table  4.22  summarizes  question 
C.2  (shortest  time),  and  Table  4.23  summarizes  question  C.3 
(fewest  personnel  for  proposal  evaluation) .  The 
conventional  approach  was  identified  as  the  longest  approach 
by  a  majority  of  the  personnel.  The  streamlined  approaches 
were  chosen  as  the  shortest  approaches  and  the  approaches 
requiring  the  fewest  number  of  people.  Table  4.24  shows  the 
results  of  survey  question  C.4  (most  deviations  from  RFP) . 
The  high  non-response  and  no  preference  rates  precluded 
further  use  of  the  results  of  this  question. 


APPROACH 


NUMBER 


PERCENT  OF  TOTAL 


CONVENTIONAL 

57 

52.3 

FOUR  STEP 

31 

28.4 

ASO  STREAMLINED 

2 

1.8 

BMO  STREAMLINED 

2 

1.8 

OTHER 

0 

0.0 

NO  PREFERENCE 

17 

15.6 

APPROACH 


NUMBER 


PERCENT  OF  TOTAL 


CONVENTIONAL 

9 

8.4 

FOUR  STEP 

14 

13.1 

ASD  STREAMLINED 

27 

25.2 

BMO  STREAMLINED 

25 

23.4 

OTHER 

13 

12.1 

NO  PREFERENCE 

19 

17.8 

TOTAL 


107 


100% 


TABLE  4.23 


Approach  Requiring  the  Fewest  Personnel  for  Proposal 

Evaluation 


APPROACH 

NUMBER 

PERCENT  OF  TOTAL 

CONVENTIONAL 

6 

5.7 

FOUR  STEP 

8 

7.6 

ASD  STREAMLINED 

32 

30.5 

BMO  STREAMLINED 

25 

23.8 

OTHER 

13 

12.4 

NO  PREFERENCE 

21 

20.0 

TOTAL 

105 

100% 

TABLE  4.24 

Approach  with 

the  Most  Deviations  from 
Criteria 

Request  for 

Proposal 

APPROACH 

NUMBER 

PERCENT 

OF  TOTAL 

CONVENTIONAL 

19 

19.4 

FOUR  STEP 

13 

13.3 

ASD  STREAMLINED 

6 

6.1 

BMO  STREAMLINED 

2 

2.0 

OTHER 

0 

0.0 

NO  PREFERENCE 

58 

59.2 

TOTAL 

98 

100% 

Section  Four  -  Opinions  Concerning  the  Source 
Selection  Process  and  Differing  Approach  Characteristics. 
Section  four  presents  the  survey  participants  opinion  of 
various  aspects  of  the  source  selection  process  and  also  of 
the  major  identifying  characteristics  of  each  source 
selection  approach. 
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Definition  of  System  Requirements.  Table 
4.25  summarizes  the  results  of  survey  question  D.  A 


majority  of  the  survey  participants  believe  the  RFP  is 
adequately  defining  the  technical  aspects  of  the  system. 
Tables  4.26  and  4.27  indicate  a  majority  of  the  participants 


believe  the  evaluation  process  is  being  used  to  further 
define  the  system  requirements  (survey  question  H) . 


TABLE  4.25 

Does  the  RFP  Adequately  Define  the  Technical  Aspects  of  the 

System  for  the  Contractor? 


RESPONSE 


CUMBER 


PERCENT  OF  TOTAL 


35.4 

14.6 


TOTAL 


TABLE  4.26 

Is  the  Evaluation  Process  Used  to  Further  Define  the  System 

Requirements? 


RESPONSE 


NUMBER 


PERCENT  OF  TOTAL 


52.9 

47.1 


TOTAL 


TABLE  4.27 


Approach  Most  Used  to  Further  Define  System  Requirements 


APPROACH  NUMBER  PERCENT  OF  TOTAL 


CONVENTIONAL 

14 

19.2 

FOUR  STEP 

5 

6.8 

ASD  STREAMLINED 

0 

0.0 

BMO  STREAMLINED 

1 

1.4 

ALL  APPROACHES 

ARE  USED 

34 

46.6 

DON'T  KNOW/ 

OTHER 

19 

26.1 

TOTAL 

73 

100% 

Identifying 

Characteristics  of 

Source 

Selection  Approaches.  Tables  4.28  thru  4.32  contain  the 
participants  opinions  on  whether  attempts  to  modify  the 
source  selection  process  are  advantageous  or  not.  The 
modifications  presented  in  the  survey  include  putting  a  page 
limit  on  the  contractor's  proposal  (question  E) ,  having  a 
contractor  presentation  (question  F) ,  evaluating  the 
technical  and  cost  proposals  separately  (question  G) , 
limiting  the  number  of  people  on  the  SSEB  (question  I) , 
combining  the  SSEB  and  SSAC  (question  J) ,  and  establishing  a 
time  limit  (question  K) .  Only  the  separate  evaluation  of 
the  technical  and  cost  proposals  and  the  time  limit  were 
viewed  as  being  not  advantageous  to  the  source  selection 
process  by  a  majority  of  the  participants. 


RESPONSE 


NUMBER 


PERCENT  OF  TOTAL 


YES 

117 

86.7 

NO 

18 

13.3 

TOTAL 

135 

103% 

After 

TABLE  4. 

the  Competitive  Range  Has 
Contractor  Presentation 

29 

Been  Established,  would  a 
be  an  Advantage? 

RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

YES 

83 

63.4 

NO 

48 

36.6 

TOTAL 

131 

100% 

TABLE  4.30 

Is  Evaluating  Cost  and  Technical  Proposals  Separately  an 

Advantage? 


RESPONSE 


NUMBER  PERCENT  OF  TOTAL 


^ES  47  39.8 

10  71  60.2 


TOTAL 


128 


133% 


TABLE  4.31 


Is  Limiting  the  Number  of  People  on  the  SSEB  an  Advantage? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

YES 

73 

57.5 

NO 

54 

42.5 

TOTAL 

127 

100% 

TA3LE  4.32 

Would  Combining  the  SSEB  and  SSAC  be  an  Advantage? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAi 

YES 

94 

75.8 

NO 

30 

24.2 

TOTAL 

124 

100% 

TABLE  4.33 

Is  a  Pre-established  Time  Limit  between  Proposal  Receipt  and 

Contract  Award  an  Advantage? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

YES 

57 

43.5 

NO 

74 

56.5 

TOTAL 

131 

100% 

V.  Analysis  and  Discussion 


Chapter  Overview 

This  chapter  contains  an  analysis  of  the  research 
objectives  presented  in  Chapter  I.  Each  research 
objective's  analysis  contains  the  survey  questions 
associated  with  the  objective,  the  statistical  techniques 
applied  on  the  questions,  and  the  findings. 

Research  objectives  three  thru  eight  deal  with  the 
opinions  of  experienced  source  selection  personnel 
concerning  different  aspects  of  the  source  selection 
process.  The  analysis  of  these  objectives  is  based  upon  the 
SPSS  computer  subprograms:  FREQUENCIES,  CROSSTABS,  and 
ONEWAY.  Tables  and  analysis  of  variance  were  initially 
generated  using  numerous  different  combinations  of  the 
opinion  results  and  background  information.  This  chapter 
contains  only  those  results  that  were  determined  to  be 
significant  to  the  research  effort. 

Research  Objective  1^ 

Determine  if  an  adequate  data  base  (source 
selection  records  and  personnel)  exists  and 
is  accessible  for  research. 

Data  Availability.  The  research  team  made  two 
determinations  in  response  to  research  objective  one. 

First,  the  source  selection  records  required  for  a 
comprehensive  research  study  were  incomplete  and  often 
inaccessible  for  research. 
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Secondly,  each  product  division  in  AFSC  routinely  made 


source  selections  and  had  many  individuals  experienced  in 
the  source  selection  process.  This  determination  enabled 
the  team  to  complete  the  survey  questionnaire  portion  of  the 
research  study. 

Research  Objective  2_ 

Match  a  completed  four  step  acquisition  with 
an  equivalent  conventional  acquisition  and 
determine  if  contract  relationships  can  be  used 
to  characterize  a  specific  approach's  use. 

Research  objective  two  was  not  able  to  be  met  due  to 
incomplete  and  inaccessible  four  step  source  selection 
records  as  discussed  in  Chapter  IV. 

Research  Objective  3 

Determine  if  source  selection  approach 
familiarity  plays  a  role  in  which  source 
selection  approaches  Air  Force  Systems 
Command  personnel  consider  during  the 
Source  Selection  Plan  generation. 

Survey  Questions  and  Statistical  Techniques.  CROSSTABS 
was  used  to  examine  the  relationship  between  two  survey 
questions:  Part  I,  Question  K  (approaches  considered  during 

the  SSP  generation  process)  and  Part  I,  Question  L  (source 
selection  approaches  personnel  are  familiar  with) .  An 
additional  contingency  table  was  generated  for  analysis, 
adding  product  division  as  the  third  factor. 

Source  Selection  Plan  Generation  Analysis.  Of  the  140 
people  surveyed,  117  (83.7%)  had  participated  in  an  SSP 


generation  (reference  Table  4.11).  These  results  are  broken 
out  by  product  division  as  follows:  42  -  ASD;  19  -  SD;  26  - 
ESD;  14  -  AD;  7  -  BMO;  and  9  -  other.  The  following  table 
shows  the  number  of  approaches  source  selection  personnel 
considered  during  SSP  generation: 


TABLE  5.1 

Number  of  Approaches  Considered  During  SSP  Generation 


NUMBER 


PERSONNEL 


PERCENT  OF  TOTAL 


3.4 
42.2 
29  .3 
20.7 
4.3 


TOTAL 


100% 


A  clear  majority  of  survey  participants  considered  less  than 
3  approaches  when  they  participated  in  the  SSP  generation 
process. 

Table  5.2  is  a  summarization  of  the  contingency  tables 
generated  showing  the  approaches  considered  during  SSP 
generation  by  the  approaches  source  selection  personnel  were 
familiar  with.  Tables  5.3  thru  5.6  summarize  the 
contingency  table  results  by  product  division.  The  summary 
tables  show  only  the  relationships  between  source  selection 
approaches  personnel  are  familiar  with  and  the  source 
selection  approaches  they  considered  in  SSP  generation.  The 
unfamiliar,  not  considered,  and  other  statistics  were  not 


APPROACH 


NUMBER 

FAMILIAR 


NUMBER 

CONSIDERED 


PERCENT 


L 

114 

90 

78.9 

94 

46 

48.9 

Lned 

46 

34 

73.9 

Lned 

36 

19 

52.8 

TABLE  5.3 


Summary  Table  -  Conventional  Approach 


PRODUCT 

DIVISION 


ASD 

SD 

ESD 

AD 

BMO 

Other 


NUMBER 

FAMILIAR 


NUMBER 

CONSIDERED 


PERCENT 


74.4 

94.7 

92.3 

71.4 
57.1 
55.6 


TABLE  5.4 


Summary  Table  -  Four  Step  Approach 


PRODUCT 

DIVISION 


NUMBER 

FAMILIAR 


NUMBER 

CONSIDERED 


PERCENT 


ID 

32 

15 

46.9 

* 

18 

7 

38.9 

ID 

20 

13 

65.0 

> 

11 

6 

54.5 

10 

6 

2 

33.3 

her 

7 

3 

42.9 

TABLE  5.5 


Summary  Table  -  ASD  Streamlined  Approach 


PRODUCT 

DIVISION 

NUMBER 

FAMILIAR 

NUMBER 

CONSIDERED 

PERCENT 

ASD 

34 

23 

82.4 

SD 

3 

0 

0.0 

BSD 

2 

1 

50.0 

AD 

4 

2 

50.0 

BMO 

0 

0 

0.0 

Other 

3 

3 

100.0 

TABLE  5.6 


Summary 

Table  -  BMO 

Streamlined  Approach 

PRODUCT 

DIVISION 

NUMBER 

FAMILIAR 

NUMBER 

CONSIDERED 

PERCENT 

ASD 

6 

0 

0.0 

SD 

16 

8 

50.0 

ESD 

2 

1 

50.0 

AD 

5 

3 

60.0 

3  MO 

7 

7 

100.0 

Other 

0 

0 

0.0 

Source  selection  personnel  who  had  participated  in  a 
S3P  generation  indicated  they  considered  the  conventional 
approach  78.9%  of  the  time  if  they  were  familiar  with  the 
approach.  Analyzing  the  breakout  by  product  division  shows 
that  each  product  division  considered  the  conventional 
approach  in  over  half  of  their  source  selections.  No 
product  division  appeared  to  differ  significantly  from  this 
figure. 

The  four  step  approach  was  considered  48.9%  of  the  time 


even  though  more  than  33%  of  the  people  (94  of  117)  were 


familiar  with  the  approach.  Only  BSD  and  AD  personnel 
considered  the  approach  more  than  half  of  the  time  (65%  and 
54.5%,  respectively),  while  BMO  and  SD  personnel  considered 
the  four  step  approach  33%  and  38.9%  respectively. 

3 MO  and  ASD  considered  their  streamlined  approaches  a 
majority  of  the  time  within  their  respective  product 
divisions  (ASD  -  82.4%  and  BMO  -  100%).  The  other  product 
divisions  are  limited  in  the  number  of  people  familiar  with 
either  streamlined  approach  with  the  exception  of  SD.  Space 
Division  has  a  relatively  large  number  of  people  familiar 
with  the  BMO  streamlined  approach,  but  only  half  of  the  SD 
personnel  familiar  with  the  BMO  approach  considered  it  in 
the  SSP  generation  process. 

Research  Objective  £ 

Determine  if  source  selection  personnel 
prefer  a  specific  source  selection 
approach  based  upon  an  acquisition's 
characteristics. 

Survey  Questions .  The  responses  to  eight  survey 
questions  were  reviewed  in  the  analysis  of  this  objective. 
The  survey  questions  are: 

1.  Part  II,  Question  A  -  preferred  approach. 

2.  Part  II,  Question  31  -  preferred  approach  for 
implementing  state  of  the  art  technology. 

3.  Part  II,  Question  B2  -  preferred  approach  for  using 


proven  technology. 


4.  Part  II,  Question  B3  -  preferred  approach  for  a 
contract  award  less  than  $1  million. 

5.  Part  II,  Question  B4  -  preferred  approach  for  a 
contract  award  between  $1  million  and  $5  million. 

6.  Part  II,  Question  B5  -  preferred  approach  for  a 
contract  award  greater  than  $5  million. 

7.  Part  II,  Question  B6  -  preferred  approach  when  4  or 
more  bidders  are  anticipated. 

8.  Part  II,  Question  B7  -  preferred  approach  when  less 
than  4  bidders  are  anticipated. 

Statistical  Techniques.  FREQUENCIES  generated  the 
percentage  of  occurrence  of  each  approach  within  a  question. 
Further  analysis  of  this  research  objective  required  the  use 
of  CROSSTABS  to  generate  contingency  tables  between  the 
questions  described  above  and  the  background  variable  product 
division. 

Overall  Approach  Preference .  Table  4.13  indicates  that 
50.9%  of  all  survey  participants  familiar  with  two  or  more 
source  selection  approaches  prefer  the  conventional 
approach.  Table  5.7  shows  the  crosstabulation  of  each 
product  division  with  their  source  selection  approach 
preference.  All  product  divisions  prefer  the  conventional 
approach  overall  except  ASD  and  BMO .  ASD  was  evenly  divided 
between  the  conventional  approach  and  ASD  streamlined 
approach.  BMO  was  100%  for  the  BMO  streamlined  approach. 


TABLE  5.7 


Contingency  Table  —  Preferred  Approach  by  Product  Division 


COUNT  I 
ROW  PCT  I 
COL  PCT  I 
TOT  PCT  I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

I 

BMO 

I 

OTHR 

I 

ROW 

TOT 

- 1- 

- - 

•I- 

•I- 

■I- 

I- 

I- 

— 

I 

13 

I 

13 

I 

20 

I 

5 

I 

0 

I 

3 

I 

54 

CONVEN-  I 

24.1 

I 

24.1 

I 

37.0 

I 

9.3 

I 

0 

I 

5.6 

I 

50.9 

TIONAL  I 

39.4 

I 

61.9 

I 

83.3 

I 

35.7 

I 

0 

I 

42.9 

I 

I 

12.3 

I 

12.3 

I 

18.9 

I 

4.7 

I 

0 

I 

2.8 

I 

- X- 

— 

•I- 

— 

•I- 

■I- 

■I- 

•I- 

I- 

I 

a 

I 

1 

I 

2 

I 

1 

I 

0 

I 

1 

I 

5 

FOUR  I 

a 

I 

20.0 

I 

40.0 

I 

20.0 

I 

0 

I 

20.0 

I 

4.7 

STEP  I 

a 

I 

4.8 

I 

3.3 

I 

mgs 

E9 

0 

I 

14.3 

I 

I 

0 

I 

.9 

I 

1.9 

I 

9 

0 

I 

.9 

I 

- 1- 

•I- 

•I- 

— 

•I- 

a 

■I- 

•I- 

I 

13 

I 

0 

I 

0 

I 

2 

i 

0 

I 

1 

I 

16 

ASD  I 

31.3 

I 

0 

I 

0 

I 

12.5 

i 

0 

I 

6.3 

I 

15.1 

STREAMLINEDI 

39.4 

I 

0 

I 

0 

I 

14.3 

i 

0 

I 

14.3 

I 

I 

12.3 

I 

0 

I 

0 

I 

1.9 

i 

0 

I 

.9 

I 

- x_ 

— 

•I- 

I- 

■I- 

i- 

I- 

I- 

I 

1 

I 

3 

I 

0 

I 

2 

i 

7 

I 

1 

I 

14 

BMO  I 

7.1 

I 

21.4 

I 

0 

I 

14.3 

i 

50.0 

I 

7.1 

I 

13.2 

STREAMLINEDI 

3.0 

I 

14.3 

I 

0 

I 

14.3 

1100.0 

I 

14.3 

I 

I 

.9 

I 

2.8 

I 

0 

I 

1.9 

i 

6.6 

I 

.9 

I 

- X- 

- -- 

-I- 

■I- 

I- 

i- 

I- 

■I- 

I 

2 

I 

1 

I 

1 

I 

1 

i 

0 

I 

1 

I 

6 

NO  I 

33.3 

I 

16.7 

I 

16.7 

I 

16.7 

i 

0 

I 

16.7 

I 

5.7 

PREFERENCE  I 

6.1 

I 

4.8 

I 

4.2 

I 

7.1 

i 

0 

I 

14.3 

I 

I 

1.9 

I 

.9 

I 

.9 

I 

.9 

i 

0 

I 

.9 

I 

- 1- 

— 

■I- 

I- 

I- 

i- 

I- 

I- 

I 

4 

I 

3 

I 

1 

I 

3 

i 

0 

I 

0 

I 

11 

OTHER  I 

36.4 

I 

27.3 

I 

9.1 

I 

27.3 

i 

0 

I 

0 

I 

10.4 

I 

12.1 

I 

14.3 

I 

4.2 

I 

21.4 

i 

0 

I 

0 

I 

I 

3.3 

I 

2.3 

I 

.9 

I 

2.8 

i 

0 

I 

0 

I 

- 1- 

■I- 

B 

B 

B 

B 

I- 

COLUMN 

TOTAL 

33 

31.1 

7 

6.6 

106 

100.0 

the  ccosstabulation  of  each  product  division  with  their 
choice  of  approach  for  purchasing  state-of-the-art 
technology.  Every  product  division  chose  the  conventional 

TABLE  5.8 

Contingency  Table  -  State-of-the-Ar t  by  Product  Division 


COUNT  I 
ROW  PCT  I 

COL  PCT  I  ROW 


TOT  PCT 

I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

1 

BMO 

I 

OTHR 

I 

TOT 

•I- 

— 

-I- 

— 

-I- 

-I- 

-1- 

— 

•I- 

■I- 

I 

15 

I 

14 

I 

15 

I 

6 

I 

0 

I 

5 

I 

55 

CONVEN- 

I 

27.3 

I 

25.5 

I 

27.3 

I 

10.9 

I 

0 

I 

9.1 

I 

51.4 

TIONAL 

I 

44.1 

I 

66.7 

I 

62.5 

I 

42.9 

I 

0 

I 

71.4 

I 

I 

14.0 

I 

13.1 

I 

_  T  — 

14.0 

I 

5.6 

I 

0 

I 

4.7 

I 

-  T  — 

.  ±  - 

I 

2 

•  1  ““ 

I 

3 

*  1 

I 

6 

"  1* 

I 

2 

I 

0 

•A* 

i 

1 

A 

i 

14 

FOUR 

I 

14.3 

I 

21.4 

I 

42.9 

I 

14.3 

I 

0 

i 

7.1 

i 

13.1 

STEP 

I 

5.9 

I 

14.3 

I 

25.0 

I 

14.3 

I 

0 

i 

14.3 

i 

I 

1.9 

I 

2.8 

I 

5.6 

I 

1.9 

I 

0 

i 

.9 

i 

■I- 

— 

•I- 

— 

*1- 

— - 

•I- 

— 

•I- 

— 

•i- 

-i- 

I 

10 

I 

0 

I 

0 

I 

1 

I 

0 

i 

0 

i 

11 

ASD 

I 

90.9 

I 

0 

I 

0 

I 

9.1 

I 

0 

i 

0 

i 

10.3 

STREAMLINEDI 

29.4 

I 

0 

I 

0 

I 

7.1 

I 

0 

i 

0 

i 

I 

9.3 

I 

.  T  — 

0 

I 

-  T  - 

0 

I 

.9 

I 

.  T 

0 

i 

0 

i 

.  T 

’  i."1 

I 

2 

X 

i 

2 

1 

I 

0 

’  1  *“ 

I 

2 

1 

I 

7 

*  A 

i 

1 

•  A"“ 

i 

14 

3  MO 

i 

14.3 

i 

14.3 

i 

0 

I 

14.3 

I 

50.0 

i 

7.1 

i 

13.1 

STREAMLINEDI 

5.9 

i 

9.5 

I 

0 

I 

14.3 

1100.0 

i 

14.3 

i 

i 

1.9 

i 

1.9 

i 

0 

I 

1.9 

I 

6.5 

i 

.9 

i 

■i- 

— 

•i- 

— 

-i- 

— 

•I- 

-I- 

— 

•i- 

— 

*i- 

— 

i 

3 

i 

1 

I 

3 

I 

2 

I 

0 

i 

0 

i 

9 

NO 

i 

33.3 

i 

11.1 

I 

33.3 

I 

22.2 

I 

0 

i 

0 

i 

8.4 

PREFERENCE 

i 

8.8 

i 

4.8 

I 

12.5 

I 

14.3 

I 

0 

i 

0 

i 

i 

,  T 

2.8 

i 

•  T  - 

.9 

I 

-  T  — 

2.8 

I 

•  T- 

1.9 

I 

-  T  — 

0 

i 

.  T  _ 

0 

i 

-  T  — 

*  X 

1 

2 

X 

I 

1 

A 

I 

0 

X 

i 

1 

“A  — 

i 

0 

•  A- 

I 

0 

A- 

i 

4 

OTHER 

I 

50.0 

I 

25.0 

i 

0 

i 

25.0 

i 

0 

I 

0 

i 

3.7 

I 

5.9 

I 

4.8 

I 

0 

i 

7.1 

i 

0 

I 

0 

i 

I 

1.9 

I 

.9 

I 

0 

i 

.9 

i 

0 

i 

0 

i 

•I- 

— 

•I- 

— 

-i- 

— 

■i- 

-i- 

EX 

•i- 

— 

COLUMN  34  21  24  14  7  7  107 

TOTAL  31.8  19.6  22.4  13.1  6.5  6.5  100.0 


approach  except  for  BMO  which  unanimously  chose  the  BMO 
streamlined  approach. 

Approach  Preference  for  Proven  Technology.  Table  4.15 
indicates  that  36.3%  of  survey  participants  prefer  the 
conventional  approach  for  source  selections  implementing 
proven  technology  while  35.8%  prefer  one  of  the  streamlined 
approaches.  Table  5.9  shows  the  crosstabulation  of  each 
product  division  with  their  choice  of  approach  for 
purchasing  proven  technology.  Every  product  division  except 
for  ASD  and  BMO  chose  the  conventional  approach.  A5D 
personnel  preferred  the  ASD  streamlined  approach  with  52.9% 
favoring  its  use.  BMO  personnel  were  133%  in  favor  of  the 
BMO  streamlined  approach. 

Preferred  Approach  for  Varying  Contract  Award  Costs . 
Tables  4.16,  4.17,  and  4.18  indicate  that  a  majority  of 
survey  participants  are  in  favor  of  using  a  streamlined 
approach  for  source  selections  with  a  contract  value  less 
than  five  million  dollars.  As  the  contract  award  increases 
in  value,  the  number  of  survey  participants  preferring  the 
conventional  approach  also  increases.  Table  4.18  indicates 
that  55.7%  of  survey  participants  prefer  the  conventional 
approach  for  contract  awards  greater  than  five  million 
dollars.  Table  5.10  shows  that  this  trend  is  true  for  all 
product  divisions  except  for  BMO.  The  majority  of  BMO 
personnel  prefer  the  BMO  streamlined  approach  regardless  of 
contract  award  value. 


L  0 


TABLE  5.9 


Contingency  Table  -  Proven  Technology  by  Product  Division 


COUNT  I 
ROW  PCT  I 
COL  PCT  I 
TOT  PCT  I 

ASD 

I 

SD 

I 

ESO 

I 

AD 

I 

BMO 

I 

OTHR 

I 

ROW 

TOT 

- X- 

■I- 

I- 

-  — — - 

■I- 

•I- 

■I- 

- — — - 

■I- 

—  —  —  — 

I 

5 

I 

10 

I 

17 

I 

4 

I 

0 

I 

3 

I 

39 

CONVEN-  I 

12.8 

I 

25.6 

I 

43.6 

I 

10.3 

I 

0 

I 

7.7 

I 

36.8 

TIONAL  I 

14.7 

I 

50.0 

I 

70.8 

I 

28.6 

I 

0 

I 

42.9 

I 

I 

4.7 

I 

9.4 

I 

16.0 

I 

3.8 

I 

0 

I 

2.8 

I 

- x. 

-I- 

■I- 

— 

I- 

I- 

I- 

— 

I- 

I 

2 

I 

2 

I 

3 

I 

1 

I 

0 

I 

2 

I 

10 

FOUR  I 

20.0 

I 

20.0 

I 

30.0 

I 

10.0 

I 

0 

I 

20.0 

I 

9.4 

STEP  I 

5.9 

I 

10.0 

I 

12.5 

I 

7.1 

I 

0 

I 

28.6 

I 

I 

1.9 

I 

1.9 

I 

2.8 

I 

.9 

I 

0 

I 

1.9 

I 

- 1- 

— 

-I- 

■I- 

■I- 

•I- 

■I- 

— 

•I- 

— 

I 

18 

I 

0 

I 

0 

I 

2 

I 

0 

I 

1 

I 

21 

A3D  I 

85.7 

I 

0 

I 

0 

I 

9.5 

I 

0 

I 

4.8 

I 

19.8 

STREAMLINEDI 

52.9 

I 

0 

I 

0 

I 

14.3 

I 

0 

I 

14.3 

I 

I 

17.0 

I 

0 

I 

0 

I 

1.9 

I 

0 

I 

.9 

I 

- 1- 

•I- 

.  T  - 
X 

■I- 

— 

I- 

I- 

•I- 

— 

I 

1 

I 

5 

I 

1 

I 

2 

I 

7 

I 

1 

I 

17 

BMO  I 

5.9 

I 

29.4 

I 

5.9 

I 

11.8 

I 

41.2 

I 

5.9 

I 

16.0 

STREAMLINEDI 

2.9 

I 

25.0 

I 

4.2 

I 

14.3 

1100.0 

I 

14.3 

I 

I 

.9 

I 

4.7 

I 

.9 

I 

1.9 

I 

6.6 

I 

.9 

I 

-I- 

■I- 

I- 

— - 

■I- 

I- 

— 

I- 

— 

I 

4 

I 

2 

I 

3 

I 

2 

I 

0 

I 

0 

I 

11 

NO  I 

36.4 

I 

18.2 

I 

27.3 

I 

18.2 

I 

0 

I 

0 

I 

10.4 

PREFERENCE  I 

11.8 

I 

10.0 

I 

12.5 

I 

14.3 

I 

0 

I 

0 

I 

I 

3.8 

I 

1.9 

I 

2.8 

I 

1.9 

I 

0 

I 

0 

I 

- 1- 

— 

•I- 

■I- 

I- 

■I- 

■I- 

I- 

I 

4 

I 

1 

I 

0 

I 

3 

I 

0 

I 

0 

I 

8 

OTHER  I 

50.0 

I 

12.5 

I 

0 

I 

37.5 

I 

0 

I 

0 

I 

7.5 

I 

11.8 

I 

5.0 

I 

0 

I 

21.4 

I 

0 

I 

0 

I 

I 

3.8 

I 

.9 

I 

0 

I 

2.8 

I 

0 

I 

0 

I 

- 1- 

— 

■I- 

■I- 

— 

I- 

— 

■I- 

•I- 

— 

■I- 

COLUMN 

TOTAL 

34 

32.1 

20 

13.9 

24 

22.6 

14 

13.2 

7 

6.6 

i 

7 

6.6 

i 

106 

100.0 

Preferred  Approach  for  Varying  Number  of  Bidders . 

Tables  4.19  and  4.20  indicate  that  a  majority  of  source 

selection  personnel  prefer  the  conventional  approach 

regardless  of  the  number  of  bidders.  Tables  5.11  and  5.12 

show  that  the  conventional  approach  is  slightly  more  favored 

5-11 


TABLE  5.10 


Contingency  Table  -  Contract  Cost  by  Product  Division 


COUNT  I 

ROW  PCT  I 

COL  PCT  I 

ROW 

TOT  PCT  I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

I 

BMO 

I 

OTHR 

I 

TOT 

- 1- 

-I- 

-I- 

-I- 

•I- 

— 

— 

*1- 

— 

-I- 

— 

I 

16 

I 

14 

I 

19 

I 

7 

I 

0 

I 

3 

I 

59 

CONVEN-  I 

27.1 

I 

23.7 

I 

32.2 

I 

11.9 

I 

0 

I 

5.1 

I 

55.7 

TIONAL  I 

47.1 

I 

70.0 

I 

79.2 

I 

50.0 

I 

0 

I 

42  9 

I 

I 

15.1 

I 

13.2 

I 

17.9 

I 

6.6 

I 

0 

I 

2.8 

I 

- 1- 

-I- 

— 

-I- 

•I- 

-I- 

— 

— 

*1- 

-I- 

I 

0 

I 

2 

I 

1 

I 

1 

I 

0 

I 

1 

I 

5 

FOUR  I 

0 

I 

40.0 

I 

20.0 

I 

20.0 

I 

0 

I 

20.0 

I 

4.7 

STEP  I 

0 

I 

10.0 

I 

4.2 

I 

7.1 

I 

0 

X 

14.3 

I 

I 

0 

I 

1.9 

I 

.9 

I 

.9 

I 

0 

I 

2.8 

I 

- X- 

•I- 

-I- 

-I- 

— 

-I- 

— 

— 

-I- 

-I- 

— 

I 

9 

I 

0 

I 

0 

I 

1 

I 

0 

I 

1 

I 

11 

ASD  I 

81.8 

I 

0 

I 

0 

I 

9.1 

I 

0 

I 

9.1 

I 

10.4 

STREAMLINEDI 

26.5 

I 

0 

I 

0 

I 

7.1 

I 

0 

I 

14.4 

I 

I 

8.5 

I 

0 

I 

0 

I 

.9 

I 

0 

I 

.9 

I 

- X- 

-I- 

-I- 

— 

-I- 

— 

-I- 

— 

— 

-I- 

-I- 

I 

2 

I 

1 

I 

0 

I 

2 

I 

7 

I 

1 

I 

13 

BMO  I 

15.4 

I 

7.7 

I 

0 

I 

15.4 

I 

53 

.8 

I 

7.7 

I 

12.3 

STREAMLINEDI 

5.9 

I 

5.0 

I 

0 

I 

14.3 

1100 

.0 

I 

14.3 

I 

I 

1.9 

I 

.9 

I 

0 

I 

1.9 

I 

6 

.6 

I 

.9 

I 

- X- 

-I- 

•I- 

-I- 

. — - — 

-I- 

— 

— 

-I- 

•I- 

I 

1 

X 

1 

I 

2 

I 

0 

I 

0 

I 

0 

I 

4 

NO  I 

25.0 

I 

25.0 

I 

50.0 

I 

0 

I 

0 

I 

0 

X 

3.8 

PREFERENCE  I 

2.9 

I 

5.0 

I 

8.3 

I 

0 

I 

0 

I 

0 

I 

I 

.9 

I 

.9 

I 

1.9 

I 

0 

I 

0 

X 

0 

I 

- X- 

-I- 

— 

-I- 

— 

-I- 

-I- 

— 

— 

■I- 

— 

-I- 

I 

6 

I 

2 

I 

2 

I 

3 

I 

0 

I 

1 

I 

14 

OTHER  I 

42.9 

I 

14.3 

I 

14.3 

I 

21.4 

I 

0 

I 

7.1 

I 

13.2 

I 

17.6 

X 

10.0 

I 

8.3 

I 

21.4 

I 

0 

I 

14.3 

I 

I 

5.7 

I 

1.9 

I 

1.9 

I 

2.8 

I 

0 

I 

.9 

I 

OTHER 


TABLE  5.11 


Contingency  Table  -  Less  than  4  Bidders  by  Product  Division 


COUNT 

I 

ROW  PCT 

I 

COL  PCT 

I 

ROW 

TOT  PCT 

I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

I 

BMO 

I 

OTHR 

I 

TOT 

-I- 

— 

-I- 

— 

-I- 

-I- 

-I- 

— 

— 

-I- 

-I- 

— 

I 

10 

I 

10 

I 

18 

I 

7 

I 

0 

I 

3 

I 

48 

CONVEN- 

I 

20.8 

I 

20.8 

I 

37.5 

I 

14.6 

I 

0 

I 

6.3 

I 

45.7 

TIONAL 

I 

30.3 

I 

47.6 

I 

75.0 

I 

53.8 

I 

0 

I 

42.9 

I 

I 

9.5 

I 

9.5 

I 

17.1 

I 

6.7 

I 

0 

I 

2.9 

I 

-I- 

-I- 

-I- 

-I- 

-I- 

— 

— 

-I- 

-I- 

I 

0 

I 

2 

I 

4 

I 

0 

I 

0 

I 

1 

I 

7 

FOUR 

I 

I 

28.6 

I 

57.1 

I 

n 

’d 

t- 

X 

m 

U 

-r 

X 

14.3 

T 

4. 

c  n 

w  •  • 

STEP 

I 

0 

I 

9.5 

I 

16.7 

I 

0 

I 

0 

I 

14.3 

I 

I 

0 

I 

1.9 

I 

3.8 

I 

0 

I 

0 

I 

1.0 

I 

-I- 

— 

-I- 

— 

•I* 

-I- 

-I- 

— 

— 

-I- 

•I- 

- - 

I 

13 

I 

0 

I 

0 

I 

2 

I 

0 

I 

1 

I 

16 

ASD 

I 

81.3 

I 

0 

I 

0 

I 

12.5 

I 

0 

I 

6.3 

I 

15.2 

STREAMLINEDI 

39.4 

I 

0 

I 

0 

I 

15.4 

I 

0 

I 

14.3 

I 

I 

12.4 

I 

0 

I 

0 

I 

1.9 

I 

0 

I 

12.4 

I 

•I- 

■I- 

-I- 

— 

-I- 

*1- 

— 

— 

-I- 

— 

■I- 

— 

I 

1 

I 

5 

I 

0 

I 

2 

I 

7 

I 

1 

I 

16 

3  MO 

I 

6.3 

I 

31.3 

I 

0 

I 

12.5 

I 

43 

.8 

I 

6.3 

I 

15.2 

STREAMLINEDI 

3.0 

I 

23.8 

I 

0 

I 

15.4 

1100 

.0 

I 

14.3 

I 

I 

1.0 

I 

4.8 

I 

0 

I 

1.9 

I 

6 

.7 

I 

1.0 

I 

-I- 

•I- 

-I- 

-I- 

-I- 

— 

— 

-I- 

■I- 

— 

I 

5 

I 

3 

I 

2 

I 

1 

I 

0 

I 

1 

I 

12 

NO 

I 

41.7 

I 

25.0 

I 

16.7 

I 

8.3 

I 

0 

I 

8.3 

I 

11.4 

PREFERENCE 

I 

15.2 

I 

14.3 

I 

8.3 

I 

7.7 

I 

0 

I 

14.3 

I 

I 

4.8 

I 

2.9 

I 

1.9 

I 

1.0 

I 

0 

I 

1.0 

I 

•I- 

•I- 

-I- 

— 

•I- 

— 

•I- 

— 

— 

•I- 

•I- 

•  — —  —  — 

I 

4 

I 

1 

I 

0 

I 

1 

I 

0 

I 

0 

I 

6 

OTHER 

I 

66.7 

I 

16.7 

I 

0 

I 

16.7 

I 

0 

I 

0 

I 

5.7 

I 

12.1 

I 

4.8 

I 

0 

I 

7.7 

I 

0 

I 

0 

I 

I 

3.8 

I 

1.0 

I 

0 

I 

1.0 

I 

0 

I 

0 

I 

-I- 

•I- 

— 

■I- 

-I- 

— 

•I- 

— 

— 

-I- 

— 

-I- 

...... 

COLUMN  33  21  24  13 

TOTAL  31.4  20.0  22.9  12.4 


7  105 

6.7  100.0 


6.: 


TABLE  5.12 


Contingency  Table  -  4  or  more  Bidders  by  Product  Division 


COUNT  I 
ROW  PCT  I 
COL  PCT  I 
TOT  PCT  I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

I 

BMO 

I 

OTHR 

I 

ROW 

TOT 

- 1- 

•I- 

— 

-I- 

-I- 

•I- 

■I- 

-I- 

I 

10 

I 

7 

I 

17 

I 

3 

I 

0 

I 

2 

I 

39 

CONVEN-  I 

25.6 

I 

17.9 

I 

43.6 

I 

7.7 

I 

0 

I 

5.1 

I 

37.1 

TIONAL  I 

30.3 

I 

33.3 

I 

70.8 

I 

23.1 

I 

0 

I 

28.6 

I 

I 

9.5 

1 

6.7 

I 

16.2 

I 

2.9 

I 

0 

I 

1.9 

I 

- X- 

— 

-I- 

-I- 

•I- 

— 

•I- 

■I- 

-I- 

— 

I 

2 

I 

5 

I 

4 

I 

2 

I 

0 

I 

2 

I 

15 

FOUR  I 

13.3 

I 

33.3 

I 

26.7 

I 

13.3 

I 

0 

I 

13.3 

I 

14.3 

STEP  I 

6.1 

I 

23.8 

I 

16.7 

I 

15.4 

I 

0 

I 

28.6 

I 

I 

1.9 

I 

4.8 

I 

3.8 

I 

1.9 

I 

0 

I 

1.9 

I 

- X- 

■I- 

— 

-I- 

•I- 

■I- 

— 

•I- 

•I- 

I 

14 

I 

0 

I 

0 

I 

2 

I 

0 

I 

1 

I 

17 

ASD  I 

82.4 

I 

0 

I 

0 

I 

11.8 

I 

0 

I 

5.9 

I 

16.2 

S  TREAMLINEDI 

42.4 

I 

0 

I 

0 

I 

15.4 

I 

0 

I 

14.3 

I 

I 

13.3 

I 

0 

I 

0 

I 

1.9 

I 

0 

I 

1.0 

I 

- 1- 

■I- 

— 

-I- 

■I- 

-I- 

•I- 

-I- 

I 

1 

I 

5 

I 

0 

I 

3 

I 

7 

I 

1 

I 

17 

BMO  I 

5.9 

I 

29.4 

I 

0 

I 

17.6 

I 

41.2 

I 

5.9 

I 

16.2 

STREAMLINEDI 

3.0 

I 

23.8 

I 

0 

I 

23.1 

1100.0 

I 

14.3 

I 

I 

1.0 

I 

4.8 

I 

0 

I 

2.9 

I 

6.7 

I 

1.0 

I 

■I- 

-I- 

— 

•I- 

— 

■I- 

— 

■I- 

— 

•I- 

I 

4 

I 

3 

I 

3 

I 

1 

I 

0 

I 

1 

I 

12 

NO  I 

33.3 

I 

25.0 

1 

25.0 

I 

8.3 

I 

0 

I 

8.3 

X 

11.4 

PREFERENCE  I 

12.1 

I 

14.3 

I 

12.5 

I 

7.7 

I 

0 

I 

14.3 

I 

I 

3.8 

I 

2.9 

I 

2.9 

I 

1.0 

I 

0 

I 

1.0 

I 

- 1- 

— - - 

•I- 

■  —  —  —  —  - 

-I- 

-  — —  — - 

-I- 

- 

•I- 

-I- 

■  —  —  —  —  - 

-I- 

-  —  — —  — 

I 

2 

I 

1 

I 

0 

I 

2 

I 

0 

I 

0 

I 

5 

OTHER  I 

40.0 

I 

20.0 

I 

0 

I 

40.0 

I 

0 

I 

0 

I 

4.8 

I 

6.1 

I 

4.8 

I 

0 

I 

15.4 

I 

0 

I 

0 

I 

I 

1.9 

I 

1.0 

I 

0 

I 

1.9 

I 

0 

I 

0 

I 

- 1- 

— 

■I- 

-I- 

■I- 

•I- 

>1- 

— 

-I- 

— - 

COLUMN 

TOTAL 

33 

31.4 

21 

20.0 

24 

22.9 

13 

12.4 

7 

6.7 

7 

6.7 

105 

100.0 

Research  Objective  5 

Determine  if  source  selection  personnel 
have  preconceived  notions  about  the  time 
and  manpower  requirements  for  a  given 
source  selection  approach. 


Survey  Questions .  Three  survey  questions  were  used  in 
the  analysis  of  this  objective.  The  survey  questions  are: 

1.  Part  II/  Question  Cl  -  approach  with  the  longest 
period  of  time  between  receipt  of  proposal  and  contract  award; 

2.  Part  II,  Question  C2  -  approach  with  the  shortest 
period  of  time  between  receipt  of  proposal  and  contract  award; 

3.  Part  II,  Question  C3  -  approach  with  the  fewest 
number  of  people  for  proposal  evaluation. 

Statistical  Techniques .  The  statistical  techniques  used 
in  the  analysis  of  this  objective  were  the  same  as  those  used 
for  Research  Objective  4. 

Time  Requirements  Analysis.  Only  109  survey  participants 
responded  to  this  set  of  questions.  Table  4.21  shows  that 
most  survey  participants  (52,3%)  believe  the  conventional 
approach  requires  the  longest  period  of  time  from  RFP  release 
to  contract  award;  28.4%  believe  the  four  step  takes  the 
longest.  Table  5.13  shows  this  response  trend  is  not  true 
across  product  divisions.  SO  and  BMO  appear  to  differ 
significantly  from  the  overall  response.  A  majority  of  SD 
respondents  (47.6%)  believe  the  four  step  approach  takes  the 
longest  time.  BMO  is  equally  split  between  the  four  step  and 
conventional  approaches. 

The  majority  of  the  survey  participants  (60.7%)  were 
split  between  the  streamlined  approaches  and  "other" 
approaches  for  the  approach  requiring  the  shortest  period  of 
time  (reference  Table  4.22).  As  expected,  the  contingency 
table  for  shortest  time  by  product  division  (Table  5.14) 
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TABLE  5.13 


Contingency  Table  -  Longest  Time  by  Product  Division 


COUNT  I 
ROW  PCT  I 

COL  PCT  I  ROW 

TOT  PCT  I  ASD  I  SD  I  ESD  I  AD  I  BMO  I  OTHR  I  TOT 

24  I  31  12  I  71  31  31  57 

CONVEN-  I  42.1  I  14.0  I  21.1  I  12.3  I  5.3  I  5.3  I  52.3 

TIONAL  I  66.7  I  38.1  I  50.0  I  50.0  I  42.9  I  42.9  I 

I  22.0  I  7.3  I  11.0  I  6.4  I  2.8  I  2.8  I 

—  _ _  T  _ _ T _ -  =  T  _ T _ — 

I  71  10  I  61  31  31  21  31 

FOUR  I  22.6  I  32.3  I  19.4  I  9.7  I  9.7  I  6.5  I  28.4 

STEP  I  19.4  I  47.6  I  25.0  I  21.4  I  42.9  I  28.6  I 


ASD  1100.0  I 

STREAMLINEDI  5.6  I 

I  1.8  1 


I 

BMO  I 

STREAMLINEDI 


0  I  50.0  I 
01  4.8  I 


0  I  50.0  I 
0  I  14.3  I 


NO  I 

17.6  I 

11 

.8 

I  35.3 

I  23.5  I 

0 

I  11.8 

I  15.6 

PREFERENCE  I 

8.3  I 

9 

.5 

I  25.0 

I  28.6  I 

0 

I  28.6 

I 

I 

2.8  I 

1 

.8 

I  5.5 

I  3.7  I 

-I - 1- 

0 

I  1.8 

•  I - 

I 

-X - 

COLUMN 

36 

21 

24 

14 

7 

7 

109 

TOTAL 

33.0 

19 

.3 

22.0 

12.8 

6.4 

6.4 

100.0 

reveals  that  the  majority  of  ASD  and  BMO  personnel  picked  their 
own  product  division's  streamlined  approach  as  being  the 
shortest.  SD  also  strongly  supported  the  BMO  streamlined 
approach  as  the  shortest  approach.  ESD  and  AD  had  no  clear 
choice  for  the  shortest  approach. 


TABLE  5.14 


Contingency  Table  -  Shortest  Time  by  Product  Division 


COUNT  I 
ROW  PCT  I 

COL  PCT  I  ROW 


TOT  PCT 

I 

ASD 

I 

SD 

I 

ESD 

i 

AD 

I 

BMO 

I 

OTHR 

I 

TOT 

-I- 

— 

•I- 

■I- 

— 

-i- 

— 

■I- 

•I- 

— 

*1 

I 

0 

I 

2 

I 

7 

i 

0 

I 

0 

I 

0 

I 

9 

CONVEN- 

I 

0 

I 

22.2 

I 

77.8 

i 

0 

I 

0 

I 

0 

I 

8.4 

TIONAL 

I 

0 

I 

10.5 

I 

29.2 

i 

0 

I 

0 

i 

0 

I 

T 

0 

T 

1  Q 

T 

6_5 

i 

0 

I 

0 

i 

0 

I 

•I- 

■I- 

—  - - 

■i- 

■i- 

— 

■I- 

— 

•I- 

■I- 

— 

I 

4 

I 

1 

i 

7 

i 

2 

I 

0 

I 

0 

I 

14 

FOUR 

I 

28.6 

i 

7.1 

i 

50.0 

i 

14.3 

I 

0 

I 

0 

I 

13.1 

STEP 

I 

11.1 

i 

5.3 

i 

29.2 

i 

14.3 

I 

0 

I 

0 

I 

I 

3.7 

i 

.9 

i 

6.5 

i 

1.9 

I 

0 

I 

0 

I 

•I- 

— 

■I- 

[ 

-i- 

•I- 

•i- 

— 

-I- 

I 

22 

i 

0 

i 

2 

I 

0 

I 

2 

I 

27 

ASD 

I 

81.5 

i 

0 

i 

7.4 

I 

0 

I 

7.4 

I 

25.2 

STREAMLINEDI 

61.1 

i 

0 

I 

14.3 

I 

0 

i 

28.6 

I 

I 

20.6 

i 

0 

i 

1.9 

I 

0 

i 

1.9 

I 

•I- 

— 

-I- 

— 

-i- 

•I- 

•1- 

— 

-I- 

I 

2 

I 

12 

i 

1 

I 

2 

I 

7 

i 

1 

I 

25 

BMO 

I 

8.0 

I 

48.0 

i 

4.0 

I 

8.0 

I 

23.0 

i 

4.0 

I 

23.4 

STREAMLINEDI 

5.6 

I 

63.2 

i 

4.2 

I 

14.3 

1100.0 

I 

14.3 

I 

I 

1.9 

I 

11.2 

i 

.9 

I 

1.9 

I 

6.5 

i 

.9 

I 

-I- 

— 

•I* 

— 

-i- 

•i- 

-I- 

•I* 

■I- 

I 

2 

i 

1 

i 

a 

i 

5 

I 

0 

I 

3 

I 

19 

NO 

I 

10.5 

i 

5.3 

i 

42.1 

i 

26.3 

I 

0 

i 

15.8 

I 

17.8 

PREFERENCE 

I 

5.6 

i 

5.3 

i 

33.3 

i 

35.7 

I 

0 

I 

42.9 

I 

I 

1.9 

I 

.9 

I 

7.5 

i 

4.7 

I 

0 

i 

2.8 

I 

•I- 

•I- 

•i- 

*i- 

— 

•I- 

■I- 

■I- 

I 

6 

i 

2 

i 

1 

i 

3 

I 

0 

i 

1 

I 

13 

OTHER 

I 

46.2 

I 

15.4 

i 

7.7 

i 

23.1 

I 

0 

i 

7.7 

I 

12.1 

I 

16.7 

i 

10.5 

i 

4.2 

i 

21.4 

I 

0 

I 

14.3 

I 

I 

5.6 

i 

1.9 

i 

.9 

i 

2.8 

I 

0 

i 

.9 

I 

COLUMN 

36 

•  x  ■ 

19 

“1  “ 

mm 

HO 

"1“ 

7 

"1* 

107 

TOTAL 

33.6 

17.8 

US 1 

6.5 

100.0 

Manpower  Requirements  Analysis.  Table  4.23  shows  that 
the  survey  participants  were  evenly  divided  between  the  ASD 
streamlined  approach  (30.5%),  BMO  streamlined  approach 
(23.8%),  and  no  preference  (20%)  for  the  approach  requiring 


the  fewest  number  of  people.  Table  5.15  indicates  that  ASD 
and  BMO  personnel  strongly  favored  their  respective  product 
division's  streamlined  approach.  SD  strongly  supported  the 
BMO  streamlined  approach  and  ESD  did  not  respond  favorably  to 
either  streamlined  approach.  Most  ESD  personnel  chose  either 
the  conventional  approach,  the  four  step  approach,  or  had  no 
preference.  Only  1  in  19  people  chose  a  streamlined  approach 
ESD  did  not  respond  favorably  to  the  streamlined  approaches 
because  only  4  people  in  ESD  were  familiar  with  one  or  more 
streamlined  approach. 

Research  Objective  6 

Determine  if  the  source  selection  process 
is  being  used  as  a  means  to  further 
define  weapon  system  requirements. 

Survey  Questions.  Survey  questions  Part  II,  D  and  H 
were  the  basis  of  this  analysis. 

Statistical  Techniques.  The  data  for  analysis  of  this 
objective  was  obtained  by  running  FREQUENCIES  and  several 
ONEWAY  analysis  of  variance  on  the  responses  to  the  above 
survey  questions  .  Two  ONEWAY  analysis  of  variance  tables 
were  generated  using  the  responses  to  the  previously 
described  questions  as  the  dependent  variables  and  product 
division  as  the  independent  variable.  We  were  looking  for 
any  significant  difference  of  opinion  between  the  product 
divisions  using  a  95%  confidence  interval  (significant 
difference  if  F-probability  is  less  than  .05) . 


TABLE  5.15 
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[■. 

i- 


k 


Contingency  Table  -  Manpower  by  Product  Division 


COUNT  I 
ROW  PCT  I 
COL  PCT  I 
TOT  PCT  I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

I 

BMO 

I 

OTHR 

I 

ROW 

TOT 

- 1- 

— 

-I- 

— 

-I- 

— 

-I- 

■I- 

— 

■I- 

•I- 

I 

0 

I 

0 

I 

6 

I 

0 

I 

0 

I 

0 

I 

6 

CONVEN-  I 

0 

I 

0 

1100.0 

I 

0 

I 

0 

I 

0 

I 

5.7 

TIONAL  I 

0 

I 

0 

I 

26.1 

I 

0 

I 

0 

I 

0 

I 

I 

0 

I 

0 

I 

5.7 

I 

0 

I 

0 

I 

0 

I 

- X- 

— 

-I- 

-I- 

-I- 

-I- 

•I- 

•I- 

I 

0 

I 

1 

I 

7 

I 

0 

I 

0 

I 

0 

I 

8 

FOUR  I 

0 

I 

12.5 

I 

87.5 

I 

0 

I 

0 

I 

0 

I 

7.6 

STEP  I 

0 

I 

5.3 

I 

30.4 

I 

0 

I 

0 

I 

0 

I 

I 

0 

I 

1.0 

I 

6.7 

I 

0 

I 

0 

I 

0 

I 

- X- 

- - - 

-I- 

-I- 

- - 

-I- 

— 

•I- 

— 

■I- 

-I- 

I 

23 

I 

1 

I 

1 

I 

4 

I 

1 

I 

2 

I 

32 

ASD  I 

71.9 

I 

3.1 

I 

3.1 

I 

12.5 

I 

3.1 

I 

6.3 

I 

30.5 

STREAMLINEDI 

65.7 

I 

5.3 

I 

4.3 

I 

28.6 

I 

14.3 

I 

28.6 

I 

I 

21.9 

I 

1.0 

I 

1.0 

I 

3.8 

I 

1.0 

I 

1.9 

I 

- 1- 

— 

-I- 

-I- 

•I- 

— 

•I- 

— 

■I- 

— 

•I- 

I 

1 

I 

15 

I 

0 

I 

2 

I 

6 

I 

1 

I 

25 

BMO  I 

4.0 

I 

60.0 

I 

0 

I 

8.0 

I 

24.0 

I 

4.0 

I 

23.8 

STREAMLINEDI 

2.9 

I 

78.9 

I 

0 

I 

14.3 

I 

85.7 

I 

14.3 

I 

I 

1.0 

I 

14.3 

I 

0 

I 

1.9 

I 

5.7 

I 

1.0 

I 

- 1- 

— 

-I- 

-I- 

-I- 

-I- 

-I- 

-I- 

I 

4 

I 

2 

I 

9 

I 

4 

I 

0 

I 

2 

I 

21 

NO  I 

19.0 

I 

9.5 

I 

42.9 

I 

19.0 

I 

0 

I 

9.5 

I 

20.0 

PREFERENCE  I 

11.4 

I 

10.5 

I 

39.1 

I 

28.6 

I 

0 

I 

28.6 

I 

I 

3.8 

I 

1.9 

I 

8.6 

I 

3.8 

I 

0 

I 

1.9 

I 

- 1- 

-I- 

-I- 

-  - - —  - 

-I- 

■I- 

-I- 

■I- 

I 

7 

I 

0 

I 

0 

I 

4 

I 

0 

I 

2 

I 

13 

OTHER  I 

53.3 

I 

0 

I 

0 

I 

30.8 

I 

0 

I 

15.4 

I 

12.4 

I 

20.0 

I 

0 

I 

0 

I 

28.6 

I 

0 

I 

28.6 

I 

I 

6.7 

I 

0 

I 

0 

I 

3.8 

I 

0 

I 

1.9 

I 

- 1- 

— 

-I- 

-I- 

-I- 

■I- 

■I- 

- - 

-I- 

COLUMN 

TOTAL 

35 

33.3 

19 

18.1 

23 

21.9 

14 

13.3 

7 

6.7 

7 

6.7 

105 

100.0 

Request  For  Proposal  Technical  Definition.  A  majority 
of  survey  participants  (85.4%)  indicated  that  RFPs 
adequately  define  the  technical  aspects  of  a  weapon  system 
for  the  contractor  (Table  4.25).  The  analysis  of  variance 
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I 

f 

shows  no  statistical  difference  between  the  product 
|  divisions  responses  in  this  area  (see  ANOVA  Table  C-l, 

Appendix  C) . 

Requirements  Definition  in  the  Source  Selection 

■  Process .  The  results  in  Table  4.29  indicate  that  a  small 
majority  (52.9%)  of  survey  participants  believe  the  source 
selection  evaluation  process  is  used  to  further  define 

j  system  requirements.  When  the  survey  responses  are  broken 

down  by  product  division,  an  analysis  of  variance  shows  no 
statistical  difference  exists  between  the  product  divisions 

■  (reference  ANOVA  Table  C-2,  Appendix  C) . 

The  results  in  Table  4.30  indicate  that  72.7%  of  the 
survey  participants  believing  that  the  source  selection 
,  process  is  used  to  further  define  system  requirements 

either  did  not  know  which  approach  was  used  more 
prevalently  to  define  system  requirements  or  thought  that 
all  approaches  were  used. 

Research  Objective  1_ 

Determine  the  perceptions  of  source 
selection  personnel  towards  the  major 
identifying  characteristics  of  each 
source  selection  approach. 

Survey  Questions .  There  were  six  survey  questions 
associated  with  this  objective.  They  are: 

1.  Part  II,  Question  E  -  putting  a  page  limit  on  a 
contractor's  proposal; 
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2.  Part  II,  Question  F  -  having  contractor  presentations 
after  initial  proposal  evaluations; 

3.  Part  II,  Question  G  -  having  the  technical  and  cost 
proposals  submitted  separately; 

4.  Part  II,  Question  I  -  limiting  the  number  of  people 
on  the  SSEB; 

5.  Part  II,  Question  J  -  combining  the  SSEB  and  SSAC; 

and 

6.  Part  II,  Question  K  -  having  a  pre-established  time 
limit  on  the  evaluation  process. 

Statistical  Techniques.  FREQUENCIES  generated  the 
percentage  of  Yes  and  No  responses  for  each  question.  ONEWAY 
analysis  of  variance  generated  additional  analysis  data  to 
determine  if  there  were  any  differences  of  opinion  between 
product  divisions  (independent  variables).  The  responses  of 
at  least  two  product  divisions  would  be  considered 
statistically  different  if  the  F-probability  was  less  than  .05 
(95%  confidence  interval) .  ONEWAY  does  not  tell  which  two 
product  divisions  differ  in  their  responses,  only  that  some 
significant  difference  exists.  Therefore,  the  Tukey  post  hoc 
contrast  test  was  used  for  this  purpose. 

Page  Limit  on  Contractor's  Proposal  Analysis.  A  majority 
of  the  respondents  (83.6%)  considered  a  page  limit  to  be  an 
advantage  in  the  proposal  evaluation  process  (reference  Table 
4.26).  The  results  of  the  ONEWAY  analysis  of  variance 
indicate  that  there  is  no  significant  difference  between 


product  division  responses  (reference  ANOVA  Table  C-3, 
Appendix  C) . 


Contractor  Presentation  Analysis.  Table  4.27  shows  that 
63.4%  of  the  survey  participants  favored  a  contractor 
presentation  after  the  competitive  range  had  been  established. 
ONEWAY  results  show  no  significant  difference  in  opinion 
between  product  divisions  (reference  ANOVA  Table  C-4,  Appendix 
C)  . 

Separate  Evaluation  of  Technical  and  Cost  Proposals 
Analysis.  Most  responses  to  the  question  were  generally 
negative.  A  majority  (60.2%)  of  survey  participants  do  not 
believe  evaluating  the  technical  and  cost  proposals  separately 
is  an  advantage  in  the  evaluation  process.  There  was  no 
significant  difference  in  opinion  between  the  product 
divisions  on  this  question  (reference  ANOVA  Table  C-5, 

Appendix  C) . 

Limiting  the  Number  of  People  on  the  SSEB  Analysis .  A 
slight  majority  (57.5%)  of  the  respondents  consider  limiting 
the  number  of  people  on  the  SSEB  to  be  an  advantage  in  the 
evaluation  process.  ONEWAY  results  indicate  that  product 
divisions  differ  significantly  in  their  response  to  this 
question  (F-probability  less  than  .05)  (reference  ANOVA  Table 
C-6,  Appendix  C) . 

The  Tukey  post  hoc  contrast  test  for  systematically 
comparing  all  possible  pairs  of  product  division  responses 
reveals  that  the  difference  in  responses  identified  by  the 


ONEWAY  analysis  occurs  between  ASO  and  BMO.  Table  5.16  shows 
that  BMO  is  100%  in  favor  of  limiting  the  number  of  people  on 
the  SSEB  while  58.3%  of  the  ASO  responses  are  against  the 
limitation. 

TABLE  5.16 

Contingency  Table  -  Limit  Number  of  People  on  SSEB  by 

Product  Division 


COUNT  I 
ROW  PCT  I 

COL  PCT  I  ROW 

TOT  PCT  I  ASD  I  SO  I  ESD  I  AD  I  BMO  I  OTHR  I  TOT 

- 1 - X - X - 1 - 1 - 1 - 1 - 

I  20  I  14  I  18  I  6  1  7  1  8  1  73 

YES  I  27.4  I  19.2  I  24.7  I  8.2  I  9.6  I  11.0  I  57.5 

I  41.7  I  77.8  I  60.0  I  46.2  1100.0  I  72.7  I 

I  15.7  I  11.0  I  14.2  I  4.7  I  5.5  I  6.3  I 

- 1 - 1 - 1 - X - 1 - 1 - 1 - 

I  28  I  41  12  I  71  01  31  54 

NO  I  51.9  I  7.4  I  22.2  I  13.0  I  0  I  5.6  I  42.5 

I  58.3  I  22.2  I  40.0  I  53.8  I  01  27.3  I 

I  22.0  I  3.1  I  9.4  I  5.5  I  01  2.4  I 

- 1 - 1 - 1 - X - 1 - 1 - 1 - 

COLUMN  48  18  30  13  7  11  127 

TOTAL  37.8  14.2  23.6  10.2  5.5  8.7  100.0 


Combining  the  SSEB  and  SSAC  Analysis.  Table  4.32  shows 
that  a  majority  (75.8%)  consider  combining  the  SSEB  and  SSAC 
functions  an  advantage  in  the  source  selection  process.  The 
ONEWAY  analysis  of  variance  indicates  that  there  is  no 
significant  difference  of  opinion  between  product  divisions  on 
this  question  (reference  ANOVA  Table  C-7 ,  Appendix  C) . 

Pre-established  Time  Limit  Analysis.  A  majority  (56.5%) 
of  survey  participants  do  not  consider  a  pre-established  time 
limit  between  proposal  receipt  and  contract  award  an 


advantage,  but  the  analysis  of  variance  reveals  that  at  least 
two  product  divisions  differ  significantly  in  opinion  on  this 
issue  (reference  ANOVA  Table  C-8,  Appendix  C) . 

The  Tukey  post  hoc  contrast  test  indicates  that  BMO 
responses  differ  significantly  from  both  SD  and  BSD.  Table 
5.17  reveals  that  BMO  is  100%  in  favor  of  a  time  limit  while 
71.4%  of  SD  personnel  and  73.3%  of  ESD  personnel  do  not  think 
a  time  limit  is  an  advantage  in  the  source  selection  process. 

TABLE  5.17 

Contingency  Table  -  Time  Limit  by  Product  Division 


COUNT  I 
ROW  PCT  I 

COL  PCT  I  ROW 

TOT  PCT  I  ASD  I  SD  I  ESD  I  AD  I  BMO  I  OTHR  I  TOT 

l  23  1  61  81  61  71  71  57 

YES  I  40.4  I  10.5  I  14,0  I  10.5  I  12.3  I  12.3  I  43.5 

X  48.9  I  28.6  I  26.7  I  42.9  1100.0  I  58.3  I 
I  17.6  I  4.6  I  6.1  I  4.6  I  5.3  I  5.3  I 

- X - X - 1 - 1 - X - 1 - 1 - 

I  24  I  15  I  22  I  81  01  51  74 

NO  I  32.4  I  20.3  I  29.7  I  10.8  I  0  I  6.8  I  56.5 

I  51.1  I  71.4  I  73.3  I  57.1  I  01  41.7  I 

I  13.3  I  11.5  I  16.8  I  6.1  I  01  3.8  I 

COLUMN  47  21  30  14  7  12  131 

TOTAL  35.9  16.0  22.9  10.7  5.3  9.2  100.0 


Research  Objective  j3 

Determine  what  training  source  selection 
personnel  receive  in  the  area  of  source 
selection. 

Survey  Questions .  Three  survey  question  responses 
were  analyzed  for  this  objective.  The  survey  questions 


1.  Part  I ,  Question  H  -  formal  Air  Force  training 


courses  attended; 

2.  Part  I ,  Question  I  -  training  prior  to  first  source 
selection;  and 

3.  Part  I,  Question  J  -  bulk  of  source  selection 
experience. 

Statistical  Techniques,  initially  the  relative 
frequencies  of  the  above  questions  were  analyzed.  In 
addition,  CROSSTASS  was  used  to  generate  additional  data  for 
analysis  showing  the  relationships  between  the  above  survey 
responses  and  the  respondent's  product  division. 

Training  Prior  to  First  Source  Selection 
Participation.  Table  4.10  indicates  that  89%  of  survey 
participants  did  not  receive  source  selection  training 
prior  to  participating  in  their  first  source  selection. 

Table  5.18  shows  that  this  trend  holds  true  across  all  the 
product  divisions. 

Military  Source  Selection  Training.  Table  4.9 
indicates  that  35.7%  of  survey  participants  have  never  had 
any  formal  Air  Force  training  courses  covering  the  source 
selection  process.  Table  5.19  shows  that  ASD  has  the 
smallest  percentage  of  personnel  (29.4%)  who  have  not 
attended  a  formal  course  while  BMO  has  the  largest 
percentage  (57.1%). 


TABLE  5.18 


TABLE  5.19 


Contingency  Table  -  Formal  Training  by  Product  Division 


COUNT 

I 

ROW  PCT 

I 

COL  PCT 

I 

ROW 

TOT  PCT 

I 

ASD 

I 

SD 

I 

ESD 

I 

AD 

I 

BMO 

I 

OTHR 

I 

TOT 

- 1- 

-I- 

— 

-I- 

-I- 

-I- 

-I- 

■I- 

I 

15 

I 

8 

I 

12 

I 

7 

I 

4 

I 

4 

I 

53 

NO 

I 

30.0 

I 

16.0 

I 

24.0 

I 

14.0 

I 

8.0 

I 

8.0 

I 

35.7 

COURSES 

I 

29.4 

I 

38.1 

I 

35.3 

I 

50.0 

I 

57.1 

I 

30.8 

I 

I 

10.7 

I 

5.7 

I 

8.6 

I 

5.0 

I 

2.9 

I 

2.9 

I 

I- 

— 

-I- 

-I- 

— 

-I- 

— 

■I- 

•I- 

•I- 

I 

13 

I 

8 

I 

13 

I 

3 

I 

1 

I 

3 

I 

41 

ONE 

I 

31.7 

I 

19.5 

I 

31.7 

I 

7.3 

I 

2.4 

I 

7.3 

I 

29.3 

COURSE 

I 

25.5 

I 

38.1 

I 

38.2 

I 

21.4 

I 

14.3 

I 

23.1 

I 

I 

9.3 

I 

5.7 

I 

9.3 

I 

2.1 

I 

.7 

I 

2.1 

I 

-I- 

-I- 

• - 

-I- 

— 

-I- 

-I- 

— 

•I- 

■I> 

I 

15 

I 

1 

I 

6 

I 

4 

I 

1 

I 

6 

I 

33 

TWO 

I 

45.5 

I 

3.0 

I 

18.2 

I 

12.1 

I 

3.0 

I 

18.2 

I 

23.6 

COURSES 

1 

29.4 

I 

4.8 

I 

17.6 

I 

28.6 

I 

14.3 

I 

46.2 

I 

I 

10.7 

I 

.7 

I 

4.3 

I 

2.9 

I 

.7 

I 

4.3 

I 

•I- 

— 

-I- 

— 

-I- 

-I- 

•I- 

— 

•I- 

•I- 

I 

8 

I 

3 

I 

3 

I 

0 

I 

1 

I 

0 

I 

15 

THREE 

I 

53.3 

X 

20.0 

I 

20.0 

I 

0 

I 

6.7 

I 

0 

I 

10.7 

COURSES 

I 

15.7 

I 

14.3 

I 

8.8 

I 

0 

I 

14.3 

I 

0 

I 

I 

5.7 

I 

2.1 

I 

2.1 

I 

0 

I 

.7 

X 

0 

I 

•I- 

-I- 

— 

-I- 

— - 

-I- 

-I- 

— — - 

■I- 

— 

■I- 

I 

0 

I 

1 

I 

0 

I 

0 

I 

0 

I 

0 

I 

1 

FOUR 

I 

0 

1100.0 

I 

0 

I 

0 

I 

0 

I 

0 

I 

.7 

COURSES 

I 

0 

I 

4.8 

I 

0 

I 

0 

X 

0 

I 

0 

I 

I 

0 

I 

.7 

I 

0 

I 

0 

I 

0 

I 

0 

I 

COLUMN 

’I- 

1 

51 

“I* 

* 

34 

•1“ 

14 

"l-" 

7 

•1  " 

13 

•1“ 

140 

TOTAL 

36.4 

15.0 

24.3 

10.0 

5.0 

9.3 

100.0 

TABLE  5.20 


Contingency  Table  -  Source  Selection  Experience  by  Product 

Division 


COUNT  I 
ROW  PCT  I 

COL  PCT  I  ROW 


TOT  PCT 

I 

ASD 

I  SD 

I 

ESD 

I 

AD 

I 

8  MO 

I  OTHR 

I  TOT 

•I- 

— 

-I - 

— 

-I- 

— 

-I- 

-I- 

-I - 

•I - 

I 

0 

I 

0 

I 

0 

I 

0 

I 

0 

I  1 

I  1 

FORMAL 

I 

0 

I 

0 

I 

0 

I 

0 

I 

0 

1100.0 

I  .7 

TRAINING 

I 

0 

I 

0 

I 

0 

I 

0 

I 

0 

I  7.7 

I 

I 

0 

I 

0 

I 

0 

I 

0 

I 

0 

I  .7 

I 

I  42  I  12  I  23  I  81  41  10  I  99 


ON-THE-JOB  I 

42.4 

I 

12.1 

I 

23.2 

I 

8.1 

I 

4.0 

I 

10.1 

I 

71.2 

TRAINING  I 

84.0 

I 

57.1 

I 

67.6 

I 

57.1 

I 

57.1 

I 

76.9 

I 

I 

30.2 

I 

8.6 

I 

16.5 

I 

5.8 

I 

2.9 

I 

7.2 

I 

- I- 

•I- 

— 

■I- 

— 

•I- 

— 

•I- 

— 

•I- 

— 

•I 

I 

6 

I 

8 

I 

9 

I 

6 

I 

3 

I 

2 

I 

34 

INDIVIDUAL  I 

17.6 

I 

23.5 

I 

26.5 

I 

17.6 

I 

8.8 

I 

5.9 

I 

24.5 

STUDY  I 

12.0 

I 

38.1 

I 

26.5 

I 

42.9 

I 

42.9 

I 

15.4 

I 

I 

4.3 

I 

5.8 

I 

6.5 

I 

4.3 

I 

2.2 

I 

1.4 

I 

- 1- 

—  —  —  —  —  — 

•I- 

■I- 

— 

•I- 

-I- 

•I- 

•I- 

I 

3 

I 

0 

I 

2 

I 

0 

I 

0 

I 

0 

I 

2 

NOT  I 

0 

I 

0 

1100.0 

I 

0 

I 

0 

I 

0 

I 

1.4 

EXPERIENCED I 

0 

I 

0 

I 

5.9 

I 

0 

I 

0 

I 

0 

I 

I 

0 

I 

0 

I 

1.4 

I 

0 

I 

0 

I 

0 

I 

- !- 

•I- 

•I- 

-I- 

— 

■I- 

- X- 

— 

•I- 

I 

2 

I 

1 

I 

0 

I 

0 

I 

0 

I 

0 

I 

3 

OTHER  I 

66.7 

I 

33.3 

I 

0 

I 

0 

I 

0 

I 

0 

I 

2.2 

I 

4.0 

I 

4.8 

I 

0 

I 

0 

I 

0 

I 

0 

I 

I 

1.4 

I 

.7 

I 

0 

I 

0 

I 

0 

I 

0 

I 

- X- 

COLUMN 

TOTAL 

50 

36.0 

•I- 

21 

15.1 

■I- 

34 

24.5 

-I- 

14 

10.1 

•I- 

7 

5.0 

■I- 

13 

9.4 

•I- 

139 

L00.0 

VI.  Conclusions  and  Recommendations 


Chapter  Overview 

This  chapter  summarizes  the  conclusions  that  can  be 
drawn  from  this  study  on  the  source  selection  process.  This 
study's  large  sample  size  and  high  survey  questionnaire 
response  rate  indicate  that  conclusions  made  for  the  sample 
of  source  selection  personnel  should  be  indicative  of  the 
entire  Air  Force  source  selection  population. 

The  chapter  is  broken  into  three  sections  -  Training, 
Preferred  Approaches  and  Identifying  Characteristics,  and 
the  Request  For  Proposal.  Each  section  contains  conclusions 
that  can  be  drawn,  a  discussion  ot  the  conclusions,  and 
recommendations. 

Training 

Conclusions.  The  following  is  a  list  of  the 
conclusions  drawn  from  the  research  data  concerning  source 
selection  training: 

1.  One  of  the  most  significant  conclusions  which 
can  be  drawn  from  the  data  is  that  Air  Force  personnel  are 
not  receiving  adequate  training  prior  to  source  selection 
participation.  Only  11  percent  of  Air  Force  personnel 
surveyed  receive  training  before  participating  in  their 
first  source  selection. 

2.  Formal  Air  Force  training  courses  are  not 
covering  the  source  selection  process  in  enough  detail. 


Thirty-six  percent  of  source  selection  personnel  have  never 
attended  formal  training.  Personnel  gain  the  bulk  of  their 
source  selection  experience  from  on-the-job  training  and 
self  study. 

Discussion.  The  lack  of  source  selection  training  has 
a  definite  impact  on  the  source  selection  process.  Comments 
from  source  selection  personnel  contained  in  Appendix  D 
indicate  that  the  evaluation  process  frequently  gets  bogged 
down  due  to  inexperienced  and  unqualified  personnel.  Source 
selection  personnel  need  training  in  areas  ranging  from  how 
to  fill  out  basic  source  selection  forms  to  what  to  look  for 
daring  the  evaluation.  Personnel  working  in  source 
selections  have  totally  different  concepts  of  what  the 
source  selection  process  is  intended  to  do,  how  the  process 
is  expected  to  work,  and  what  their  role  in  the  process  is. 
Some  personnel  perceive  source  selections  as  something  to  be 
avoided  due  to  this  mystery  and  the  uncertainty  surrounding 
them. 

Limited  source  selection  training  also  affects  SSP 
generation.  The  lack  of  exposure  to  different  types  of 
approaches,  especially  the  streamlined  approaches,  limits  a 
program  managers  options  when  deciding  which  source 
selection  approach  is  best  suited  for  a  particular  program. 
Program  managers  need  to  be  familiar  with  all  source 
selection  approaches  in  order  to  choose  or  tailor  an 
approach  to  meet  their  specific  needs.  The  results  indicate 


that  this  is  especially  true  in  product  divisions  with 
streamlined  approaches.  The  results  indicate  a  strong  bias 
towards  consideration  of  streamlined  approaches  during  SSP 
generation  when  program  managers  were  familiar  with  them. 

Recommendations .  The  following  is  a  list  of 
recommendations  which  could  improve  the  training  of  Air 
Force  source  selection  personnel: 

1.  A  comprehensive  training  course  should  be 
established  to  cover  all  aspects  of  the  source  selection 
process.  The  course  design  should  include  source  selection 
plan  preparation,  basic  source  selection  forms,  and  basic 
Air  Force  source  selection  philosophy.  The  course  should 
provide  an  understanding  of  all  source  selection  approaches 
and  give  personnel  the  information  needed  to  be  more 
creative  in  tailoring  the  source  selection  process  to  a 
particular  program.  This  course  could  be  an  expansion  of  an 
already  existing  Professional  Continuing  Education  course  or 
be  a  separate  new  course. 

2.  A  short  training  course  should  be  established 
locally  at  each  product  division.  This  course  should  train 
new  personnel  in  the  overall  concepts  of  source  selection 
and  the  basic  mechanics  of  the  evaluation/review  process. 

The  course  would  familiarize  the  personnel  on  the  intended 
purpose  of  the  process  and  what  is  expected  of  them  during 
the  course  of  the  evaluation. 
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Preferred  Approaches  And  Identifying  Characteristics 


Conclusions.  The  following  is  a  list  of  the 
conclusions  drawn  from  the  research  data  concerning 
preferred  approaches  under  varying  conditions  and  preferred 
approach  characteristics: 

1.  Even  though  a  majority  of  source  selection 
personnel  identified  the  conventional  approach  as  taking  the 
longest  time  from  RFP  release  to  contract  award,  it  is  still 
the  preferred  approach  overall.  The  conventional  approach 
is  preferred  for  source  selections  implementing  state-of- 
the-art  technology  and  for  contracts  costing  more  than  5 
million  dollars.  For  source  selections  involving  smaller 
dollar  figures,  the  streamlined  approaches  are  preferred. 

The  results  indicate  that  as  the  complexity  and  cost  of 
programs  increase,  personnel  have  a  greater  preference  for 
the  conventional  approach. 

2.  Experienced  source  selection  personnel  do  not 
have  a  high  opinion  of  the  four  step  approach.  Source 
selection  personnel  consistently  preferred  the  conventional 
approach  over  the  four  step  approach  under  all  conditions. 
Less  than  5  percent  of  source  selection  personnel  preferred 
the  four  step  overall. 

3.  Source  selection  personnel  are  generally 
receptive  to  attempts  to  streamline  the  source  selection 
process.  The  comments  from  source  selection  personnel 
contained  in  Appendix  D  indicate  that  people  are  generally 


I 

I 

> 

frustrated  with  the  time  required  to  complete  a  source 

selection  evaluation. 

I 

Discussion.  The  following  is  a  discussion  of  the 
conclusions  listed  above: 

^  1.  There  are  several  interesting  trends 

associated  with  conclusion  1.  As  the  dollar  value  of  the 
contract  rises,  a  shift  occurs  from  the  number  of  people 
choosing  the  streamlined  approach  to  the  conventional 
approach.  In  the  product  divisions  without  a  streamlined 
approach,  a  much  higher  percentage  of  personnel  chose  the 
conventional  approach  as  the  preferred  approach  for  state- 
of-the-art  technology  and  high  cost  programs.  Also,  ESD  and 
SD  personnel  chose  the  conventional  approach  for  proven 

,  technology  programs  as  well.  ASD  and  AD  personnel  follow  a 

different  trend.  These  product  divisions  prefer  a 
streamlined  approach  for  proven  technology  and  the 
conventional  approach  for  state-of-the-art  technology. 
Finally,  BMO  personnel  are  very  strong  advocates  of  their 
streamlined  approach  and  consider  it  the  most  appropriate 
approach  for  any  program. 

2.  The  four  step  approach  is  not  considered 
appropriate  by  more  than  15  percent  of  the  source  selection 
personnel  in  any  of  the  given  categories  (cost,  number  of 
bidders,  or  level  of  technology) .  The  reason  is  not  that 
the  personnel  do  not  know  about  the  approach  because  more 
than  70  percent  replied  they  were  familiar  with  it.  Most 


6-5 


people  (60%)  felt  that  evaluating  the  technical  and  cost 
proposals  separately  was  not  an  advantage  to  the  evaluation 
process.  The  negative  comments  received  from  this  question 
ran  along  two  lines:  1)  the  separate  evaluations  lengthen 
the  source  selection  process;  or  2)  cost/benefit  tradeoff 
analysis  can't  be  performed  on  the  different  proposals. 

3.  Of  the  six  identifying  characteristics 
included  in  the  survey  (questions  EfF,G,I,J,K  -  Part  II), 
four  were  chosen  by  a  majority  of  the  personnel  as  being 
advantageous  to  the  source  selection  process.  Eighty-six 
percent  of  the  survey  participants  feel  that  putting  a  page 
limit  on  the  contractor's  proposals  will:  1)  speed  the 
evaluation  process;  2)  cut  down  on  the  extraneous  material 
contractors  include  in  the  proposals;  and  3)  force  the 
contractors  to  think  through  the  proposals  prior  to 
preparing  them.  This  characteristic  received  the  greatest 
support  from  source  selection  personnel. 

Two  other  areas  also  received  strong  support  from 
source  selection  personnel.  They  are:  1)  a  contractor 
presentation  with  question  and  answer  period  after  the 
competitive  range  is  established;  and  2)  combining  the 
SSEB/SSAC  functions.  Limiting  the  number  of  people  on  the 
SSEQ  was  also  identified  as  advantageous  by  a  majority  of 
the  personnel  but  many  people  expressed  concern  in  the 


comments  about  the  qualifications  of  the  evaluators. 


Surprisingly,  most  people  did  not  favor  a  time  limit  on 
the  evaluation  process.  The  general  feeling  was  that  there 


were  too  many  complexities  and  unexpected  problems  that 
could  not  be  accounted  for  in  a  time  limit. 

Recommendations .  The  comments  included  in  Appendix  D 
should  be  thoroughly  reviewed  for  additional  ideas  for 
improving  the  source  selection  process.  After  this  review, 
another  survey  should  be  administered  to  experienced  source 
selection  personnel  to  gather  their  opinions  on  these  ideas. 
Finally,  the  results  of  this  survey  should  be  evaluated  by  a 
panel  of  highly  experienced  source  selection  personnel  from 
all  the  product  divisions  and  an  AFSC-wide  streamlined 
approach  should  be  established. 

The  Request  For  Proposal 

Conclusions.  Only  one  conclusion  can  be  drawn  from  the 
research  data  concerning  the  RFP .  The  conclusion  is  that 
Air  Force  source  selection  personnel  feel  the  RFP  adequately 
defines  the  technical  aspects  of  a  system  for  the 
contractor.  Eighty-five  percent  of  source  selection 
personnel  agree  with  this  statement. 

Discussion.  Source  selection  personnel  feel  that  the 
RFP  adequately  defines  the  technical  requirements  of  a 
system,  but  more  than  half  of  the  personnel  (53%)  believe 
these  requirements  are  further  defined  in  the  evaluation 
process.  The  evaluation  process  is  not  designed  to  further 
define  system  requirements.  The  use  of  it  for  this  purpose 


indicates  that  even  though  personnel  believe  they  are  doing 
a  good  job  defining  the  technical  aspects,  a  great  deal  of 
uncertainty  still  exists. 

Recommendations .  The  following  is  a  list  of 
recommendations  for  areas  of  further  study  concerning  the 
RFP: 

1.  Conduct  a  survey  of  major  weapon  system  and 
subsystem  contractors  to  assess  their  opinion  of  the 
adequacy  of  the  technical  requirement  definitions  contained 
in  the  RFP.  A  comparison  could  then  be  made  between 
contractor  opinions  and  the  opinions  of  Air  Force  source 
selection  personnel. 

2.  Conduct  a  comparison  between  the  technical 
requirements  specified  in  RFPs  and  the  technical 
requirements  specified  on  actual  contracts.  This  would 
allow  the  assessment  of  the  actual  changes  made  to  the 
technical  requirements  during  the  evaluation  process. 
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Appendix  A:  Survey  Cover  Letter 


LS  (Capt  Hugo/Capt  Gray,  AV  785-6569) 
subject  Source  Selection  Survey 


to  Survey  Participant 


1.  Presently,  the  Air  Force  is  being  scrutinized  by  both 
Congress  and  the  news  media  as  to  how  wisely  we  are  spending  the 
taxpayers'  money.  A  main  focus  of  this  scrutiny  is  the  Air  Force 
source  selection  process.  You  have  been  identified  through 
coordination  with  ASD/PMPS  as  having  valuable  experience  in  this 
area  which  can  contribute  significantly  to  our  AFIT  source 
selection  research  project. 

2.  The  attached  survey  measures  your  perceptions  and  attitudes 
concerning  the  source  selection  process.  The  data  collected  will 
be  used  to  compile  a  statistical  analysis  to  aid  in  planning 
future  source  selections.  The  survey  has  been  coordinated 
through  the  Military  Personnel  Center  Survey  Control  Office,  the 
Civilian  Personnel  Office,  Air  Force  Systems  Command  Civilian 
Personnel  Office,  and  the  local  labor  unions  at  each  product 
division.  The  survey  control  number  is  85-42. 

3.  Please  take  the  time  to  complete  the  attached  questionnaire 
and  return  it  to  us  in  the  enclosed  envelope  within  7  days  of 
receipt.  Your  individual  response  will  be  combined  with  other 
responses  and  will  not  be  attributed  to  you  personally. 

4.  Your  participation  is  completely  voluntary,  but  we  certainly 
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Appendix  B:  Survey  Instrument 


SURVEY  REGARDING  SOURCE  SELECTIONS  IN  AIR  FORCE  SYSTEMS 

COMMAND 


This  survey  will  take  approximately  15  minutes  to 
complete.  The  survey  contains  two  sections.  The  first 
section  collects  background  and  experience  information  which 
will  be  used  to  group  your  opinions  from  the  second  section. 
Your  responses  will  greatly  help  in  our  analysis  of  the 
different  approaches  used  in  the  source  selection  process. 
For  your  convenience  a  summary  of  each  of  the  different 
source  selection  approaches  is  attached  at  the  end  of  the 
survey.  Please  feel  free  to  make  additional  comments  as  you 
fill  out  the  survey.  THANK  YOU  FOR  YOUR  TIME  &  COOPERATION. 


PART  I  Background  Information: 

A.  Military  Rank  or  Civilian  Grade: 

B.  Office  Symbol  _ 

C.  AFSC  Product  Division 


o  ASD 
o  SD 
o  ESD 


O  AD 
O  BMO 

o  other  (specify  here 


D.  Type  of  Procurement  Most  Familiar  with: 

o  Aircraft  o  Armament 

o  Electronics  o  Space/Missile 

o  other  (specify  here  _ )  . 

E.  Number  of  each  type  of  Source  Selection  you  have 
participated  in: 


Conventional  _ .  ASD  Streamlined 

4-step  _ .  BMO  Streamlined 

o  None 

Other  (Specify  type  and  number _ 

Current  Functional  Area 

o  Program/Project  Management 
o  Contracting/Manufacturing  Management 
o  Engineering 
o  Conf iguration  Management 
o  Program  Control/Business  Management 
o  Logistics 
o  Test  and  Evaluation 
o  Other  (Please  specify 


Education  (highest  degree  awarded) 


o  High  School  Diploma  o  Masters  Degree 

o  Associate  Degree  o  Doctorate  Degree 

o  Bachelors  Degree 

What  formal  Air  Force  training  courses  have  you 
attended  that  covered  source  selections: 

o  None 

o  Systems  100 
o  Systems  200 

o  PPM  152  Contract  Administration 
o  LOG  224  Logistics  Management 
o  PPM  302  Government  Contract  Law 
o  Other  (Please  specify  _ ) 

Prior  to  participating  in  your  first  source 
selection,  did  you  receive  any  formal  source 
selection  training? 

o  Yes  o  No  o  Have  never  participated 

From  what  source  did  you  get  the  bulk  of  your  source 
selection  experience: 

o  Formal  Air  Force  training  courses 
o  On-the-job  training  (OJT) 
o  Individual  Study 

o  Not  experienced  in  the  source  selection  process 
o  Other  (Please  specify _ )  . 

Have  you  ever  participated  in  a  Source  Selection 
Plan  (SSP)  generation? 

o  Yes  o  No 

If  yes,  which  approaches  were  considered  as 
alternatives  during  the  SSP  generation  process? 

o  Conventional  o  ASD  Streamlined 

o  4-step  o  BMO  Streamlined 

o  Other  (Please  specify  _ )  . 

Please  check  all  the  source  selection  approaches 
you  are  familiar  with,  even  if  you  have  never  used 
them. 

o  Conventional  o  ASD  Streamlined 

o  4-step  o  BMO  Streamlined 

o  None 

o  Other  (Please  specify  _ )  . 


If  you  are  familiar  with  at  least  2  of  the  source 
selection  approaches,  please  continue  with  Question  A. 
OTHERWISE,  skip  to  page  5,  question  D  and  continue. 
THANK  YOU  III 

PART  II  Opinions  —  Source  Selection  Approaches: 

NOTE:  Please  pick  only  one  approach  for  each  question 

A.  Which  approach  do  you  prefer? 


o  Conventional 
o  4-stap 
o  None 

o  Other  (Please  specify 


o  ASD  Streamlined 
o  BMO  Streamlined 


B.  Which  approach  would  you  use: 

1.  for  a  system  implementing  state-of-the-art 
technology? 


o  Conventional 
o  4-step 
o  No  preference 
o  Other  (Please  specify 


o  ASD  Streamlined 
o  BMO  Streamlined 


for  a  system  using  proven  technology? 


o  Conventional 
o  4-step 
o  No  preference 
o  Other  (Please  specify 


o  ASD  Streamlined 
o  BMO  Streamlined 


for  an  anticipated  contract  award  less  than 

$1,000,000? 


o  Conventional 
o  4-step 
o  No  preference 
o  Other  (Please  specify 


o  ASD  Streamlined 
o  BMO  Streamlined 


for  an  anticipated  contract  award  between 
$1,000,000  and  $5,000,000? 


o  Conventional 
o  4-step 
o  No  preference 
o  Other  (Please  specify 


o  ASD  Streamlined 
o  BMO  Streamlined 


•,  \  * 
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5. 

for  an  anticipated  contract 

award  greater  than 

$5 

,000 ,000? 

o 

Conventional 

o 

ASD  Streamlined 

o 

4-step 

o 

BMO  Streamlined 

o 

No  preference 

o 

Other  (Please  specify  _ 

) 

6. 

when  4  or  more  bidders  are  anticipated? 

o 

Conventional 

o 

ASD  Streamlined 

o 

4-step 

o 

BMO  Streamlined 

o 

No  preference 

o 

Other  (Please  specify  _ 

) 

7. 

when  less  than  4  bidders  are 

s  anticipated? 

0 

Conventional 

o 

ASD  Streamlined 

o 

4-step 

o 

BMO  Streamlined 

o 

No  preference 

o 

Other  (Please  specify  _ 

)  . 

Given 

similar  programs  of  equal 

complexity  and 

equal 

cost,  which  source  selection 

approach  would 

require  s 

1. 

the  longest  period  of  time  between  receipt  of 

proposal  and  contract  award? 

o 

Conventional 

o 

ASD  Streamlined 

o 

4-step 

o 

BMO  Streamlined 

o 

No  opinion 

o 

Other  (Please  specify  _ 

)  . 

2. 

the  shortest  period  of  time 

between  receipt  of 

proposal  and  contract  award? 

o 

Conventional 

o 

ASD  Streamlined 

o 

4-step 

o 

BMO  Streamlined 

o 

No  opinion 

o 

Other  (Please  specify  _ 

)  . 

3. 

the  fewest  number  of  people 

foe 

■  proposal 

evaluation? 

0 

Conventional 

o 

ASD  Streamlined 

o 

4-step 

0 

BMO  Streamlined 

o 

No  opinion 

0 

Other  (Please  specify  _ 

)  . 

4.  the  most  deviations  from  the  Request  for 
Proposal  (RFP)  criteria? 

o  Conventional  o  A3D  Streamlined 

o  4-step  o  BMO  Streamlined 

o  No  opinion 

o  Other  (Please  specify  _ )  . 

D.  In  general,  does  the  Request  for  Proposal 
adequately  define  the  technical  aspects  of  the 
system  for  the  contractor? 

o  Yes  o  No 

E.  Is  putting  a  page  limit  on  a  contractor's  proposal 
an  advantage  in  the  evaluation  process? 

o  Yes  o  No 

Why? 


F.  After  the  competitive  range  has  been  established, 

would  a  contractor  presentation  with  a  question  and 
answer  session  shorten  the  evaluation  process  by 
helping  to  resolve  proposal  deficiencies? 

o  Yes  o  No 

G.  The  4-step  approach  has  the  technical  and  cost 

proposals  submitted  and  evaluated  separately.  Is 
this  an  advantage? 

o  Yes  o  No 

Why? 


H.  Based  on  your  experience,  is  the  evaluation  process 
used  to  further  define  the  system  requirements? 

o  Yes  o  No 

If  yes,  under  which  approach  is  this  practice  more 
prevalent? 

o  Don't  know  o  ASD  Streamlined 

o  Conventional  o  BMO  Streamlined 

o  4-step 

o  All  approaches  are  used  for  this  practice, 
o  Other  (Please  specify  _ )  . 


I.  Would  limiting  the  number  of  people  on  the  Source 
Selection  Evaluation  Board  (SSE3)  be  an  advantage 
in  the  evaluation  process? 

o  Yes  o  No 

Why? 


J.  Would  combining  the  SSEB  and  Source  Selection 
Advisory  Council  (SSAC)  contribute  to  a  faster 
contract  award? 

o  Yes  o  No 

K.  Would  a  pre-established  time  limit  between  proposal 
receipt  and  contract  award  be  an  advantage  to  the 
proposal  evaluation  process? 

o  Yes  o  No 

Why? 


PART  III  Other  comments 


THANK  YOU  FOR  YOUR  COOPERATION 
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SUMMARY  OF  SOURCE  SELECTION  APPROACHES 


CONVENTIONAL  APPROACH 

The  Conventional  approach  establishes  a  Source  Selection 
Evaluation  Board  (SSEB)  to  evaluate  all  technical  and  cost 
aspects  of  proposals  submitted  by  contractors.  The  SSEB 
assures  that  all  proposals  meet  minimum  Government 
requirements  and  identifies  any  proposal  deficiency  to  the 
appropriate  contractor.  The  SSEB  forwards  each  contractor's 
"best  and  final  offer"  to  the  Source  Selection  Advisory 
Council  (SSAC)  where  the  documentation  and  recommendations 
of  the  SSEB  are  reviewed.  After  the  SSAC  review,  all 
recommendations  and  proposals  are  given  to  the  Source 
Selection  Authority  (SSA)  for  a  final  decision  on  contract 
award . 

ASD  STREAMLINED  APPROACH 

The  ASD  Streamlined  approach  is  a  modified  form  of  the 
Conventional  approach  where  the  Chief  of  the  SSEB  assumes 
the  responsibilities  and  tasks  belonging  to  the  SSAC  Chief 
as  well  as  his  own.  The  ASD  Streamlined  approach,  unlike 
the  Conventional  approach,  does  not  use  committees  to  do 
detailed  evaluation  and  analysis  on  each  facet  of  the 
program.  Instead,  the  SSEC  Chief  assigns  a  single  evaluator 
to  each  facet  of  the  program  who  submits  a  written 
evaluation  to  him. 

BMO  STREAMLINED  APPROACH 

The  BMO  Streamlined  approach  modifies  the  Conventional 
approach  by  restricting  the  number  of  pages  a  contractor  can 
submit  in  a  proposal  and  does  not  evaluate  any  pages  over 
the  established  maximum.  The  SSEB  is  limited  to  25 
experienced  evaluators.  These  evaluators  can  perform 
comparative  analysis  between  bid  proposals  when  the  SSEB  and 
SSAC  are  combined.  A  nine  week  standard  has  been 
established  from  proposal  receipt  to  contract  award. 
Contractor  presentations  are  allowed  after  the  competitive 
range  is  established  and  proposal  deficiencies  are  resolved. 

FOUR-STEP  APPROACH 

The  Four-Step  approach  requires  contractors  to  submit 
technical  and  cost  proposals  separately.  The  technical  and 
cost  proposals  are  evaluated  independent  of  each  other  by 
the  SSEB  and  no  deficiencies  are  discussed  with  the 
contractor.  Based  upon  these  separate  evaluations,  a  single 
contractor  is  chosen  to  enter  into  final  negotiation  with 
tne  Government.  If  the  Government  cannot  negotiate  an 
acceptable  contract  with  the  contractor  in  a  reasonable 


period  of  time,  the  Government  ceases  negotiating  and  picks 
another  contractor  for  negotiations. 


Appendix  C:  Analysis  of  Variance  (ANOVA)  Tables 

NOTE:  These  ANOVA  tables  reflect  the  SPSS  ONEWAY  results  of 

the  indicated  questions.  All  ANOVA  tables  were  generated 
using  Product  Division  as  the  independent  variable 
(grouping)  and  the  indicated  survey  questions  as  the 
dependent  variable. 


TABLE  C.l 


ANOVA 

Table  - 

RFP  Define 

System  Requirements 

SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ.  F  RATIO 

F  PR03 

BETWEEN  GROUPS 

5 

.418 

.084  .656 

.6575 

WITHIN  GROUPS 

124 

15.805 

.127 

TOTAL 

129 

16.223 

TUKEY-HSD  PROCEDURE 

—  NO  TWO  GROUPS  SIGNIFICANTLY  DIFFERENT  AT  THE  .05  LEVEL 


TABLE  C.2 


ANOVA  Table  -  Evaluation  Process  Used  to  Further  Define 

System  Requirements 


SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ. 

F  RATIO 

F  PROB 

BETWEEN  GROUPS 

5 

.893 

.179 

.704 

.6214 

WITHIN  GROUPS 

130 

32.989 

.254 

TOTAL 

135 

33.882 

TUKEY-HSD  PROCEDURE 

—  NO  TWO  GROUPS  SIGNIFICANTLY  DIFFERENT  AT  THE  .05  LEVEL 


TABLE  C .  3 


ANOVA  Table  -  Page  Limit  on  proposals 


SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ. 

F  RATIO 

F  PROB 

BETWEEN  GROUPS 

5 

.520 

.104 

.890 

.4900 

WITHIN  GROUPS 

129 

15.080 

.117 

TOTAL 

134 

15.600 

TUKEY-HSD  PROCEDURE 

—  NO  TWO  GROUPS  SIGNIFICANTLY  DIFFERENT  AT  THE  .05  LEVEL 


TABLE  C.4 

ANOVA  Table  -  Contractor  Presentation 


SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ.  F  RATIO 

F  PROB 

BETWEEN  GROUPS 

5 

2.025 

.405  1.784 

.1209 

WITHIN  GROUPS 

125 

28.387 

.227 

TOTAL 

130 

30.412 

TUKEY-HSD  PROCEDURE 

—  NO  TWO  GROUPS  SIGNIFICANTLY  DIFFERENT  AT  THE  .05  LEVEL 


TABLE  C.5 

ANOVA  Table  -  Evaluation  of  Technical  and  Cost  Proposals 

Separate 


SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ.  F  RATIO 

F  PROB 

BETWEEN  GROUPS 

5 

1.418 

.284  1.183 

.3222 

WITHIN  GROUPS 

112 

26.862 

.240 

TOTAL 

117 

28.280 

TUKEY-HSD  PROCEDURE 

—  NO  TWO  GROUPS  SIGNIFICANTLY  DIFFERENT  AT  THE  .05  LEVEL 


C-2 


TABLE  C . 6 


ANOVA 

Table  - 

Limiting  the  Number 

of  People  on  the 

SSEB 

SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ.  F  RATIO 

F  PROB 

BETWEEN 

GROUPS 

5 

3.649 

.730  3.224 

.0091 

WITHIN 

GROUPS 

121 

27.390 

.226 

TOTAL 

126 

31.039 

TUKEY-HSD  PROCEDURE 

(*)  DENOTES  PAIRS  OF  GROUPS  SIGNIFICANTLY  DIFFERENT  AT 
THE  .05  LEVEL 

G  G  G  G  G  G 
R  R  R  R  R  R 
P  P  P  P  P  P 
0  0  0  0  0  0 
0  0  0  0  0  0 
4  1  5  2  3  0 


MEAN 

GROUP 

0 

GRP004 

(BMO) 

.2222 

GRP001 

(  SD) 

.2727 

GRP005 

(OTH) 

.4000 

GRP002 

(ESD) 

.5385 

GRP003 

(  AD) 

.5833 

GRP000 

(ASD) 

TA3LE  C  .7 


ANOVA  Table  -  Combining  the  SSEB  and  SSAC 


SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ.  F  RATIO 

F  PROB 

BETWEEN  GROUPS 

5 

1.296 

.259  1.426 

.2199 

WITHIN  GROUPS 

118 

21.446 

.182 

TOTAL 

123 

22.742 

TUKEY-HSO  PROCEDURE 

—  NO  TWO  GROUPS  SIGNIFICANTLY  DIFFERENT  AT  THE  .05  LEVEL 


TABLE  C.8 


ANOVA  Table  -  Time  Limit  on  Evaluation 


SOURCE 

D.F. 

SUM  OF  SQ. 

MEAN  SQ.  F  RATIO 

F  PROB 

BETWEEN  GROUPS 

5 

3.956 

.791  3.502 

.0054 

WITHIN  GROUPS 

125 

28.242 

.226 

TOTAL 

130 

32.198 

TUKEY-HSD  PROCEDURE 

(*)  DENOTES 

PAIRS  OF 

GROUPS 

SIGNIFICANTLY  DIFFERENT  AT 

THE  .05  LEVEL 

G 

G 

G 

G 

G 

G 

R 

R 

R 

R 

R 

R 

P 

P 

P 

P 

P 

P 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

4 

5 

0 

3 

1 

2 

MEAN 

GROUP 

0 

GRP004 

(BMO) 

.4167 

GRP005 

(OTH) 

.5106 

GRP000 

(ASD) 

.5714 

GRP003 

(  AD) 

.7143 

GRP001 

(  SD) 

* 

.7333 

GRP002 

(ESD) 

* 
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Appendix  D: 


Comments  and  Responses  to  Survey  Questions 


The  comments  listed  in  this  appendix  reflect  the  views  of 
each  survey  participant  as  written  and  do  not  necessarily 
represent  the  views  of  the  authors  or  the  position  of  the 
Air  Force  Institute  of  Technology  or  the  United  States  Air 
Force. 

PART  II  Survey  Questions 

E.  Is  putting  a  page  limit  on  a  contractor's  proposal  an 
advantage  in  the  evaluation  process? 

YES  RESPONSES 

AERONAUTICAL  SYSTEMS  DIVISION 

LT  COL  —  Guide  for  the  contractor;  i.e.  Govt  emphasis, 
limits  the  time,  evaluation  must  "plow  into"  the  proposals. 
They  (KTRS)  gives  some  latitude  to  deviate  to! ! 

GS-14  —  It  puts  pressure  on  the  contractor  to 
emphasize  the  important  aspects  of  his  proposal.  "More 
weight  does  not  mean  a  better  system". 

GM-13  —  Forces  specifics,  eliminates  unnecessary  info, 
shortens  evaluation  time. 

GM-14  —  Excess  pages  are  not  looked  at  due  to  severe 
time  constraints. 

CAPT  —  Eliminates  extraneous  material  which  causes 
delays  due  to  evaluation  time  required  to  evaluate 
proposals . 


GM-15  — Insures  he  only  submits  the  most  relevant 
information  in  a  distilled  form. 

LT  COL  —  Forces  essence  to  the  top. 

GM-14  —  It  focuses  the  contractors'  attention  on  the 
important  issues  and  avoids  inclusion  of  sales  brochures. 

MAJOR  —  Places  limits  on  the  evaluation  efforts  during 
the  evaluation  process. 

G5-12  —  Reduces  source  selection  period. 

GM-15  —  Forces  discipline  on  both  govt  and  contractor. 
Govt  in  evaluation  criteria  preparation  and  contractor  in 
proposal  preparation. 
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MAJOR  —  Limits  documentation  to  facts  and  eliminates 
repetition. 

GS-12  —  Prevents  motherhood  discussions. 

GM-15  —  Forces  the  contractor  to  plan.  Reduces  govt 
evaluation  time. 

MAJOR  —  Eliminates  superfluous  info.  Tasks  respondees 
to  succintly  describe  their  technical,  cost,  &  management 
approach. 

GM-14  —  Usually  a  page  limitation  will  cut  down  the 
filler  material  some  contractors  put  in  the  RFP  and  forces 
the  contractor  to  address  the  issue  as  succinctly  as 
possible.  It  increases  CR  and  OR  activity,  but  these  can  be 
pointed  to  the  issues  at  hand. 

GM-14  —  Focuses  on  what's  important. 

GS-12  —  Cuts  out  marketing.  Contractor  forced  to  be 
clear  and  concise. 

GM-15  —  Limits  the  BS  you  have  to  sift  through. 

GM-14  —  Contractor  is  forced  to  answer  the 
questions/request  with  a  minimum  of  unnecessary  data. 

SES  —  Encourages  relevancy  of  what  is  included  and 
conciseness. 

MAJOR  —  Makes  contractor  respond  to  RFP  without  a  lot 
of  BS.  A  concise  and  to  the  point  proposal  takes  less  time 
to  evaluate. 

LT  COL  —  Less  to  reconcile. 

2LT  —  Forces  contractor  to  be  direct  and  to  the  point. 
Reduces  time  required  for  evaluation. 

MAJOR  —  Reduces  evaluation  time,  eliminates  elaborate 
plans,  approaches,  etc. 

GM-14  —  Restricts  superfluous  data. 

GS-13  —  Forces  contractor  to  be  concise  and  reduces 
extraneous  marketing  propaganda. 

LT  COL  —  Reduced  amount  of  info  to  evaluate.  Demands 
better  writing  by  proposer.  Risk  is  certain  topics  may  be 
addressed,  prompting  more  Cls/DRs. 


GM-14  —  Eliminates  sales  pitch,  gets  to  the  bottom 

line. 


SES  —  It  brings  to  focus  the  key  parameters  of  a 
proposed  design.  Otherwise,  the  contractor  prepares  volumes 
for  each  special  interest  group  in  an  attempt  to  tell  them 
what  they  think  they  want  to  see. 

MAJOR  —  Use  of  limited  resources  and  time  constraints. 

GM-13  —  Less  extraneous  material  to  read  if  pages  are 
1 imited . 

GM-13  —  Specifies  to  the  offeror  where  to  place 
emphasis. 

GS-12  —  Less  time  consuming  on  govt's  part.  Forces 
contractor  to  finalize  his  proposal  quicker,  putting  his 
response  in  fewer  words. 

GM-13  —  Helps  to  limit  extraneous  BS  the  contractor 
might  submit. 

GS-12  —  It  forces  the  offeror  to  get  the  most  out  of 
his  verbage.  Also,  it  shows  consideration  for  the  reviewer. 

CAPT  —  Contractors  will  write  to  fill  space.  Page 
limits  force  contractors  to  be  concise  and  cut  back  on 
material  that  is  not  responsive  to  the  RFP .  Page  limits  also 
save  time. 

GM-14  —  Less  material  to  evaluate;  Contractors  will 
screen  out  useless  information. 

2LT  —  Forces  offeror  to  be  concise. 

GM-13  —  Makes  the  contractor  give  us  the  concise 
information  without  superfluous  material.  Limiting  the 
number  of  pages  makes  an  evaluation  easier,  particularly 
when  there  are  seven  time  proposals  to  read  and  evaluate. 


ELECTRONIC  SYSTEMS  DIVISION 

GS-13  —  Cuts  down  on  amount  of  material  to  be 
reviewed . 

1LT  —  Limits  amount  of  reading  to  be  done  by 
evaluators. 
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1LT  —  Forces  brevity,  doesn't  swamp  evaluators  in 


data. 


GS-13  —  If  the  page  limit  is  reasonable,  and  the 
contractor  is  competent,  then  pertinent  information  can  be 
focused  and  more  quickly  evaluated. 

1LT  —  A  reasonable  page  limit  greatly  reduces  proposal 
evaluation  time  which  costs  less  for  the  government  and  the 
offerors.  This  way  only  important,  pertinent  information  is 
given.  Given  the  chance,  most  offerors  would  write  a  book 
telling  how  great  they  are  and  what  a  great  system  they  will 
deliver . 

CAPT  —  Contractor  will  hand  over  volumes  of  material 
if  allowed  to. 

GS-12  —  Eliminate  the  "BS"  and  forces  the  contractor 
to  limit  the  proposals  to  the  facts. 

GS-12  —  If  offeror  cannot  defend/define  the  proposed 
design  approach  in  the  specified  number  of  pages,  then  I 
have  low  confidence  of  the  ability  of  that  offeror  to  do  the 
job.  Also,  limits  the  amount  of  boiler  plate  and  sales 
pitches  evaluators  have  to  wade  through  to  get  to  the  design 
approach. 

GS-12  —  Contractors  with  better  understandings  of 
government  requirements  will  be  able  to  keep  their  proposals 
down  to  essential  information  without  loss  of  aspects  needed 
for  government  evaluation. 

CAPT  —  Keep  the  work/process  to  a  manageable  size. 

GS-13  —  Serves  to  streamline  process  -  less  paper  to 
read,  less  people,  etc.  Experience  is  that  with  no  limits, 
you  get  a  lot  of  marketing  PR, 

LT  COL  —  Limited  time  allowed  for  source  selection 
mandates  that  you  can  plan  your  time  for  evaluation.  You  can 
only  plan  when  you  know  the  magnitude  of  your  task. 

GS-13  —  Bounds  the  time  required  for  evaluation. 

Limits  the  offeror  to  information  specifically  requested. 

GM-13  —  Length  of  time  it  takes  to  review  proposals 
times  the  number  of  proposals. 

GM-15  —  Gives  us  a  chance  to  succeed  vs.  being  swamped 
with  meaningless  data. 

GM-14  —  Tends  to  keep  the  data  relevant. 
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GM-15  —  Limits  evaluation  time  and  potential  requests 
(by  govt  team)  for  marginal  information. 

GS-13  —  Reduces  review  time.  Does  not  allow 
contractors  to  ramble  on  in  an  effort  to  try  and  provide 
requested  information. 

GS-13  —  Reduces  the  sales  pitch. 

MAJOR  —  Forces  the  contractor  to  strictly  define  his 
proposal . 

GS-13  —  1.  Reduces  time  to  evaluate  (avoids  massive 
proposals) .  2.  Forces  succinct,  direct  proposal  responses  by 
contractors.  3.  Forces  government  to  request  just  what  it 
needs. 

GS-13  —  It  motivates  the  contractor  to  concentrate  on 
the  most  important  aspects  of  a  proposal.  Additional 
information  can  be  obtained  selectively  through 
clarification  notices. 

GM-13  —  Controls  the  display  of  bidder's  knowledge  and 
reduces  data  requiring  review. 

GM-15  —  Cuts  down  the  time  to  evaluate,  especially 
when  there  are  many  offerors. 

CAPT  —  Requires  his  thinking  what  to  prioritize  and 
enables  concise  proposals.  Saves  TET  time  in  evaluating. 

2LT  —  Makes  contractor  get  to  the  point.  Helps  to 
show  he  knows  what  he's  talking  about  if  he  can  make  it 
clear  and  concise. 

MAJOR  —  Speeds  up  the  evaluation  process. 

GM-15  —  If  reasonable  limit  -  otherwise  (if  too  few 
pages  allowed)  must  clean  up  lots  of  clarifications  thru 
negotiations . 

GM-15  —  Although  there  are  legitimate  loopholes  around 
the  page  limitation,  it  helps  to  make  the  contractor  focus 
on  what's  important. 


SPACE  DIVISION 

GM-13  —  Proposals  are  often  little  more  than  marketing 
papers  even  if  technical  designs  are  proposed.  Page  limits 
lessen  the  waste  of  preparation  costs. 


GM-13  —  Cuts  down  on  paper.  Requires  concise 
description  of  work. 

1LT  —  Doesn't  allow  extraneous  boiler  plate  to  confuse 
the  key  issues. 

GS-13  —  Since  technical  descriptions  and  requirements 
fail  to  indicate  the  necessary  information  needed  by 
offerors,  the  page  limit  allows  the  evaluation  panel  to 
prepare  CRs  and  DRs  and  eventually  get  the  information 
needed  to  make  a  selection. 

CAPT  —  It  forces  the  contractor  to  be  more  precise, 
cuts  down  evaluation  time,  and  eliminates  the  voluminous 
proposal  data  that  some  contractors  submit. 

GS-14  —  Contractors  will  write  exhaustively  if  given 
the  opportunity.  Result:  too  much  to  read. 

GS-13  —  Makes  the  contractor  take  out  unnecessary 
elaborations. 

GM-13  —  Requires  concise,  to  the  point  responses. 
Eliminates  marketing  presentations  which  have  no  merit. 

LT  COL  —  Reduces  discussions  of  less  essential 
elements  of  proposal,  increasing  evaluation  efficiency.  This 
is  an  advantage  where  the  nature  of  the  procurement  allows 
it.  However,  in  more  complex  system  acquisitions,  length 
may  be  essential  to  define  the  approach. 

CAPT  —  Forces  contractors  to  minimize  "poli tiking"  and 
only  put  concise,  pertinent  information  in  proposals. 

COL  —  Limits  irrelevant  discourse,  etc.  Speeds 
evaluation  process. 

GM-15  —  It  forces  contractors  to  focus  on  relevant 
info  rather  than  a  data  dump. 

CAPT  —  Saves  evaluation  time. 

CAPT  —  More  specific  information  provided,  less 
generic  information.  Forces  them  to  think. 

MAJOR  —  It  makes  contractor  think  through  the 
situation  and  describe  only  the  salient  factors  of  his 
proposal . 

GM-15  —  Disciplines  the  contractors  to  state  what  is 
really  important  and  reduces  evaluation  time. 
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COL  —  Forces  Govt  to  identify  decision  discriminators. 
Forces  offerors  to  concentrate  their  effort  on  these 
discriminators. 

COL  —  Reduces  contractor  cost.  Allows  evaluators  to 
read  each  proposal. 

COL  —  Focuses  attention  on  important  items. 

GM-13  —  Limits  data  to  a  manageable  and  usable  level. 
Allow  decision  making  on  key  information  rather  than  a  mass 
of  less  important  info.  Forces  both  AF  and  contractors  to 
identify  and  concentrate  on  key  issues. 

MAJOR  —  Requires  contractors  to  pay  more  attention  to 
the  PPI  identified  evaluation  criteria  and  the  ranking  of 
the  criteria.  Also,  insures  a  focus  on  the  key  important 
aspects  and  normally  eliminates  the  need  to  evaluate/read 
less  important  verbose  aspects. 

GM-14  —  Forces  contractor  to  give  more  thought  to  the 
critical  items. 

GM-14  —  Allows  evaluators  to  evaluate  entire  proposal 
in  time  allowed;  Forces  contractors  to  zero  in  on  most 
important  aspects  of  proposed  system/program. 

COL  —  You  can  tell  how  well  the  contractor  understands 
the  problem  by  how  he  selects  areas  to  emphasize.  More 
people  can  read  and  comprehend  all  of  each  proposal. 

LT  COL  —  It  puts  a  premise  on  the  contractor  thinking 
out  its  approach  before  it  writes.  It  also  allows  a  more 
detailed  analysis  by  cutting  down  the  volume  of  material  to 
read . 


ARMAMENT  DIVISION 

LT  COL  —  Reduces  evaluation  time  -  important  when 
expect  large  numbers  of  bidders. 

GM-13  —  Shortens  evaluation  time. 

GM-15  —  Requires  contractors  to  be  more  selective  and 
reduces  evaluation  time. 


GM-15 


Saves  time 


I 


Grt-13  —  It  forces  offerors  to  concentrate  on  vital 
information  that  pertains  to  the  evaluation  criteria  and  not 
submit  large  amounts  of  marketing  or  unnecessary 
information. 

GM-13  —  Saves  time  and  resources. 

:  GS-14  —  Forces  focus  on  vital  factors  -  requires 

I  understanding  of  the  problem  and  prioritization  of  relevant 

material  which  relieves  evaluator  of  perceiving  high  points 
in  shot-gun  approach. 

GM-14  —  Saves  evaluation  time.  Places  emphasis  on 
"quality  proposal"  vice  "quantity  proposal". 

I 

GM-15  —  Proposal  must  eliminate  garbage  and  go  direct 
to  substance. 


GM-15  —  Causes  him  to  concentrate  on  "area  of 
concern" . 


EASTERN  SPACE  AND  MISSILE  CENTER 

GM-13  —  Puts  a  premium  on  words. 

GM-13  —  if  a  complex  buy.  Some  contractors  have  the 
idea  that  the  longer  their  proposal  is  -  the  better  chance 
they  have  to  get  the  award. 

AERQMEDICAL  SYSTEMS  DIVISION 

MAJOR  —  Limits  extraneous  garbage  -  makes  them  think. 

LT  COL  —  Cuts  down  on  extraneous  material. 

MAJOR  —  Forces  prioritization  of  information  and 
eliminates  useless  boiler  plate. 


AIR  FORCE  AIR  LOGISTICS  CENTER 

GS-12  —  It  avoids  "hearts  and  flowers"  form  the 
contractor.  More  direct  information  is  collected. 

GS-12  —  Because  we  never  have  enough  people  to 
adequately  evaluate  the  proposals  -  more  pages=raore  people 
and  more  time. 
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GS-12  —  Forces  them  to  get  more  specific  rather  than 
ramble  on  about  general  guidance. 


NO  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 


GM-13  —  Explanation  can  become  so  condensed,  CRs  are 
required . 

GM-13  —  It  generally  results  in  requests  for  more  info 
clarification. 

GM-15  —  Not  if  we  are  trying  to  acquire  the  best 
system  that  is  the  proposal  which  best  meets  the  system  life 
cycle  requirements.  Some  argue  it  forces  the  bidder  to  be 
more  concise.  I  prefer  and  encourage  more  use  of  the  IFPP  to 
obtain  explicit  comprehensive  technical  management  and  cost 
proposals. 

GS-12  —  Limits  presentation  and  contractor's 
creativeness/ re source fulness. 

GM-14  —  There  are  ways  around  everything. 

GM-13  —  My  experience  is  that  if  the  contractor  is 
advised  that  ambiguous,  unqualified  information  is  not 
wanted  and  "procurement"  data  is  not  required.  He  provides 
good  information  in  a  reasonable  manner. 


ELECTRONIC  SYSTEMS  DIVISION 


GS-13  —  I  answered  no  because  of  the  tendency  of 
contractors  to  put  "fluff"  in  their  proposal.  However,  if  it 
is  an  extremely  complex,  state-of-the-art  source  selection, 
a  page  limitation  is  a  hinderence. 

GS-12  —  Pockets  with  additional  pages  often  used. 


SPACE  DIVISION 


GS-13  —  Would  you  buy  a  car  if  you  could  only  look  at 
it  for  20  minutes? 


ARMAMENT  DIVISION 
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GS-12  —  The  contractor  has  to  provide  additional  data 
anyway  since  he  never  (or  rarely)  gives  us  useful  data  to 
begin  with. 

GM-15  —  Large  responses  to  RFP  are  often  generated  by 
contractors  perception  of  what  we  need  rather  than  what  we 
asked  for  in  the  RFP.  Easily  circumvented  with  annexes. 


EASTERN  SPACE  AND  MISSILE  CENTER 

GM-13  —  Puts  a  limit  on  the  detail. 


AEROMEDICAL  SYSTEMS  DIVISION 


SURVEY  QUESTION 

G.  The  4-step  approach  has  the  technical  and  cost  proposals 
submitted  and  evaluated  separately.  Is  this  an  advantage? 

YES  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 

GS-14  —  For  the  standpoint  of  the  technical  evaluator, 
he  can  judge  what  is  best  to  meet  systems  requirements 
without  having  to  wander  "can  we  afford"  it.  But  after  the 
tech  eval,  I  think  the  tech  evaluator  should  review  cost 
proposals  and  help  the  cost  evaluator. 

LT  COL  —  Postpones  insertion  of  cost  bias. 

MAJOR  —  Evaluators  could  be  swayed  or  have  a  skewed 
evaluation  if  aware  of  the  costs. 

MAJOR  —  Honest  -  should  compare  against  criteria,  not 
each  other. 

GS-12  —  Allows  technical  people  to  stay  with  what  they 
know  best. 

GM-14  —  Only  good  reason  to  use  4-step  -  only  have 
prepared  and  evaluated  cost  proposals  that  have  bearing  on 
source  selection  decision. 

GS-12  —  You  don't  want  technical  evaluators  influenced 
by  cost. 

GM-15  —  Usually  allows  a  head  start  on  tech  eval. 

MAJOR  —  Prevents  leveling. 

2LT  —  Allows  technical  evaluators  to  be  uninfluenced 
by  cost. 

MAJOR  —  Evaluators  are  not  influenced  by  cost  and  cost 
analysts  are  not  confused  by  the  proposal  wording. 

GM-14  —  Technical  evaluators  are  not  influenced  by  the 
cost  estimates. 

G5-13  —  Minimizes  tech  team  bias  -  some  overlap  should 
occur  after  individual  eval  complete. 

LT  COL  —  Detailed  cost  analysis  confined  to  only  those 
in  competitive  range,  thus  reducing  workload  (at  expense  of 
timely  completion  of  overall  evaluation) . 


GM-14 


Prevents  bias 


MAJOR  —  Elimination  of  biases  early  in  the  SS  process. 

GM-13  —  Helps  insure  an  impartial  technical  review  and 
ranking . 

GS-12  —  Mgt  and  Technical  team  should  not  have  to 
consider  cost  vs.  the  proposal. 

GS-12  —  Keeps  evaluators  from  using  cost  as  the  only 
factor.  (Cost  between  contractors  may  be  close  -  yet  the 
lower  seems  to  influence  most) . 

GM-14  —  No  need  to  receive  cost  proposals  from 
offerors  not  in  the  competitive  range. 


ELECTRONIC  SYSTEMS  DIVISION 

GS-12  —  For  a  complex  acquisition,  this  approach 
allows  one  contractor  to  put  some  thought  into  his  cost 
proposal  and  allows  the  establishment  of  a  baseline  prior  to 
submission . 

GS-13  —  Decisions  and  evaluations  could  be  biased  if 
they  are  evaluated  together. 

GS-13  —  Reduces  chance  of  bias/prejudice  on  tech  eval. 

GS-12  —  Provided  the  evaluators  for  cost  and  technical 
are  different.  Reduces  potential  of  technical  evaluators 
being  influenced  by  dollar  amounts. 

GS-12  —  Evaluators  (technical)  are  not  influenced  by 
how  much  a  particular  approach  costs. 

CAPT  —  The  technical  evaluators  do  not  have  another 
selection  factor  (cost)  which  could  sway  their  decisions. 
Strictly  technical  concerns.  Leave  cost  to  the  cost  folks. 

1LT  —  The  Government  always  picks  the  lowest  bidder 
whether  or  not  he  can  do  the  job.  If  they're  technically 
incompetent,  stop  there  -  regardless  of  cost. 

1LT  —  Technical  evaluators  are  not  biased  by  costs. 

GS-12  —  Prevents  prejudisms  of  areas  which  technical 
requirements  no  met, 

2LT  —  Does  not  tie  up  a  lot  of  people. 


MAJOR  —  Concentrated  effort  can  be  expended  on  each 
submittal.  Cost  proposal  can’t  be  finalized  until  tech 
proposal  is  accomplished.  You'll  end  up  with  a  more 
realistic  cost  proposal. 

SPACE  DIVISION 

COL  —  Depends  on  situation;  offers  advantage  of 
possibly  speeding  up  the  SS  process  and  it  instill  the 
potential  for  greater  integrity,  particularly  in  Hi-tech 
areas. 

GS-12  —  Saves  time  from  evaluating  cost  proposals  of 
unacceptable  proposals. 

GS-13  —  Limits  workload  in  cases  of  many  bidders. 

CAPT  —  No  association  can  be  made. 

MAJOR  —  in  some  cases.  It  is  beneficial  when  you  are 
pushing  state-of-the-art  and  money  is  no  object. 

GM-15  —  Gives  contractors  more  time  to  prepare  cost 
proposal  which  is  entirely  (hopefully)  in  sink  with  the 
technical  proposal. 


ARMAMENT  DIVISION 

GM-15  —  Keeps  technical  and  cost  issues  separated. 

GM-15  —  Too  many  times  cost  causes  undue  technical 
compromise . 

GM-15  —  Identify  technically  acceptable  proposals 
without  cost  influence. 

GM-13  —  More  objective  evaluation  —  its  hard  to  keep 
cost  and  tech  separated  in  achieving  a  "gant"  assessed 
price. 


EASTERN  SPACE  AND  MISSILE  CENTER 
GM-13  --  Prevent  combination. 

GM-13  —  The  technical  is  not  influenced  by  cost. 


ASROMEDICAL  SYSTEMS  DIVISION 


GS-12  —  The  technical  effort  should  stand  on  its  on 
merit  and  be  judged  against  the  requirement. 


AIR  FORCE  AIR  LOGISTICS  CENTER 

GS-12  —  Forces  us  "technical  types"  to  limit  ourselves 
to  only  the  basic  technical  aspects. 


NO  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 

GM-13  —  Depends  on  what  is  being  bought.  New  R&D  -  no. 

LT  COL  —  The  conventional  does  the  same  except  that  in 
four  step  {as  I  understand  it)  you  don't  get  the  cost 
proposal  until  determined  the  technical  acceptability.  Don't 
know  the  statistics,  but  considering  that  we  do  "whole 
concept"  approach  don't  really  see  any  advantage. 

GM-13  —  Normally  technical  and  cost  are  closely  tied 
together  for  this  to  be  an  advantage  -  also,  can  cause 
longer  evaluation  period. 

GM-15  —  They  are  always  evaluated  separately  even  in  a 
conventional  mode.  Although  the  cost  panel  has  access  to  the 
technical  proposals,  no  one  except  the  cost  panel  reviews 
the  cost  proposals. 

GM-14  —  The  cost  folks  need  lots  of  technical 
assistance  to  make  sense  out  of  the  cost  proposal. 

GS-12  —  Lengthens  SS  period.  Also,  unknown  cost 
impact  on  outstanding  (unreleased)  DRs. 

GM-13  —  In  functional  areas  where  all  things  are 
equal,  the  "expert"  in  that  area  should  also  be  effective  in 
pricing. 

MAJOR  —  The  government  cannot  effectively  do 
cost/benefit  tradeoff  analyses  of  differing  technical  and 
management  approaches. 

GM-14  —  All  cost  proposals  are  submitted  and  evaluated 
separately. 

GM-15  —  This  is  absolutely  dumb  for  the  simple  reason 
that  the  cost  team  and  technical  team  must  work  together 


with  management  to  assess  and  determine  that  the  bid  cost 
relates  to  and  supports  the  technical  approach. 


GM-14  — Stretches  out  the  process  since  the  same  panels 
do  not  evaluate  both  cost  and  technical  proposals. 

MAJOR  —  In  my  areas,  the  two  proposals  are  needed  in 
order  to  evaluate  reasonableness. 

GM-13  —  So  does  conventional  and  ASD  Streamlined. 

GM-13  —  This  is  not  peculiar  to  4-step. 

GS-12  —  Although  I  have  not  evaluated  cost  proposals 
(need  to  know)  it  should  be  possible  to  evaluate  approach 
with  cost  for  same.  The  4-step  approach  is  time-consuming. 

GM-15  —  Cost  depends  on  technical  input  -  separating 
them  creates  a  myth  that  they  can  be  treated  separately. 

Too  many  neophyte  managers  really  don't  understand  what's 
going  on. 

2LT  —  I  don't  think  it  really  matters. 

GM-13  —  My  experience  shows  that  this  is  a  normal 
practice  anyway.  In  the  4-step  however  they  are  received  at 
different  times  and  could  serve  to  lengthen  the  evaluation 
process. 


ELECTRONIC  SYSTEMS  DIVISION 

GS-13  —  in  general.  Benefits  seem  to  be  all  to  the 
contractor.  He  has  time  to  determine  who  submitted  tech 
proposals  before  submitting  his  cost.  Govt  may  be  evaluating 
proposals  it  can't  afford. 

G3-13  —  The  Air  Force  can  expect  a  better  price 
through  a  best  and  final  offer  versus  a  sole-source 
negotiation. 

GM-13  —  Cost  and  technical  should  be  evaluated 
together  near  the  end  of  the  source  selection  and  can  often 
nighlight  the  excessive  cost  areas  of  the  specs  for  further 
govt  review. 

CAPT  —  Can't  get  a  balance  view  of  proposal  without 

both. 


GM-15  —  Can't  legally  establish  competitive  range 
until  evaluate  both  anyway. 


1LT  —  Technical  and  cost  should  be  delivered  together 
to  avoid  contractors  knowing  number  of  proposals.  Some 
overlap  in  evaluators  is  needed. 

GM-14  —  Tech  team  disappears  prior  to  cost  evaluation. 

GS-13  —  Conventional  is  done  essentially  the  same  way. 

SSS  —  Serial  approach  takes  too  long. 

GM-15  —  Cost  proposal  evaluation  can  be  "long  polk  in 
TEWT" . 

GM-14  —  Loss  of  technical  people  needed  to  answer  cost 
team's  questions. 

GM-15  —  We  technically  level  and  then  it's  too  late  to 
do  real  cost  analysis. 

GM-13  —  Lengthens  time  of  source  selection. 

LT  COL  —  Separating  tech  and  cost  give  contractors  a 
chance  to  play  around  too  much. 

GS-12  —  At  ESD  during  a  conventional  source  selection, 
the  cost  proposal  is  evaluated  separately?  therefore,  those 
evaluating  the  technical  proposal  are  not  influenced. 


SPACE  DIVISION 

GM-13  —  It  accomplishes  little  more  than  stretching 
out  the  process. 

CAPT  —  Soft  no.  There  needs  to  be  an  assurance  that 
the  contractor  is  considering  technical  risk  in  costs. 
Without  comparison  of  proposals,  this  can't  be  confirmed. 

GM-13  —  This  design  was  to  eliminate  bias  which  I  feel 
never  really  existed. 

GS-13  —  The  two  proposals  must  play  together  in 
determining  cost  realism. 

GS-13  —  Theoretically,  they  are  supposed  to  be 
evaluated  separately  anyway.  Comparison  of  cost  to  technical 
requirements  must  occur  eventually. 

1LT  —  For  complex  programs,  there  must  be  significant 
interplay  between  cost/technical/management  areas  to  fully 
understand  the  total  proposal  and  insure  realism  and 
credibility  of  the  cost  information. 


GM-13  —  They  should  be  evaluated  separately  in  any 
case.  Technical  evaluators  should  not  have  cost  determining 
their  evaluation  of  work  necessary  to  do  accomplish  the 
tasks. 


BALLISTIC  MISSILE  OFFICE 

MAJOR  —  Technical  weaknesses  are  sometimes  linked 
with  the  contractors  proposed  manning  and  dollars  proposed 
to  certain  SOW/WBS  tasks.  A  joint  review  of  cost/technical 
is  critical  in  establishing  the  most  probable  cost  to  the 
government. 

GM-14  —  Takes  too  much  time. 

LT  COL  —  Since  cost  is  a  statutory  evaluation  factor, 
it  is  foolish  to  segregate  an  indispensible  eval  factor  - 
How  do  you  make  an  integrated  assessment  when  an  essential 
element  is  reserved? 

COL  —  They  can  be  done  in  parallel. 


ARMAMENT  DIVISION 

GM-13  —  They  are  evaluated  separately  anyhow. 

GM-15  —  The  cost  is  based  on  the  technical  approach. 
Valuable  time  is  lost  by  the  cost  panel  waiting  for  the  cost 
proposal . 

GS-12  —  Leads  to  insufficient  evaluation  of  cost  data. 

GM-14  —  Adds  to  schedule. 

GS-14  —  They  aren't  separate.  Parts  of  the  technical 
proposal  evaluation  depend  very  much  on  the  intelligence  of 
technical-cost  tradeoffs  consistent  with  the  scheme  of  the 
RFP .  Segregation  can  lose  vital  assessment  of  a  proposal. 

GM-13  —  Every  4-step  I  have  participated  in  required 
concurrent  submission  of  technical  and  cost  proposals 
because  of  DCAA  audit  leadtime.  We  could  not  afford  the 
extra  two  or  three  weeks  due  to  separate  submittals. 

GM-15  —  Clarification  normally  does  not  make 
significant  changes  in  the  contractors  proposals. 


AEROMEDICAL  SYSTEMS  DIVISION 
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MAJOR  —  Previous  experience  shows  that  the  two  are 
inseparable  if  you  want  to  determine  who  really  knows  what 
they  are  talking  about. 

MAJOR  —  The  early  filter  could  invite  early  protests. 
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SURVEY  QUESTION 

I.  Would  limiting  the  number  of  people  on  the  Source 
Selection  Evaluation  Board  (S3EB)  be  an  advantage  in  the 
evaluation  process? 


YES  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 

GM-13  —  Currently  too  many  manhours  are  wasted  in  an 
area  where  ideas  are  battered  around. 

LT  COL  —  Saves  time. 

MAJOR  —  Provided  qualified  experienced  personnel  are 
used  the  time  for  proposal  evaluation  could  be  significantly 
reduced . 

LT  COL  —  If  right  people  i.e.  experienced  ,  know  the 
SS  process.  Unfortunately,  we  are  not  enjoying  this  now. 

GM-13  —  Inverse  relationship  between  the  number  of 
people  and  the  number  of  timely  decisions  made. 

GM-15  —  Deliberate  limiting  of  personnel  would  imply 
acceptance  of  a  reduced  quality  product.  This  is  BMO's 
approach. 

GM-14  —  It  is  possible  to  have  too  many  opinions  when 
trying  to  reach  a  decision. 

GM-15  —  Numerous  evaluators  drive  evaluation  criteria 
and  standards  to  the  "nit"  level  with  commensurate  CRs,  DRs , 
etc.  which  in  turn  extend  the  process. 

GM-14  —  Definitely.  We  spend  most  of  our  time  trying 
to  calculate  and  understand  the  evaluators  who  are  normally 
inexperienced . 

GM-15  —  People  generate  work  for  each  other. 

GM-14  —  If  experienced  personnel  assigned.  I 
personally  believe  that  too  many  unnecessary  requirements 
changes  are  introduced  because  too  many  people  are  involved. 

GS-13  —  If  these  people  are  truly  experts  and  only 
"evaluate".  After  evaluation  who  checks  to  assure  we  are 
awarding  a  good  contract? 
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3M-15  —  I  believe  whats  more  important  is  limiting  the 
number  of  people  by  experience  and  qualifications.  We  need 
to  develop  a  strong  ASD  source  selection  capability.  Many 
companies  establish  proposal  writing  departments  and 
proposal  organizations  whict  specialize  in  this  critical 
activity.  ASD  needs  to  build  our  expertise  in  this  area. 

LT  COL  —  Less  learning  and  confusion. 

MAJOR  —  Minimum  essential,  qualified  evaluators  could 
expedite  evaluation  time. 

GM-13  —  Shortens  time. 

GM-13  —  If  they  are  qualified  and  dedicated  to  the 
source  selection. 

GS-12  —  Concentrate  on  quality  -  not  quantity.  That  is 
why  I  prefer  BMO  streamlined  approach. 

GM-14  —  Cut  down  on  the  time  in  source  selection. 
Generates  fewer  unnecessary  DRs ,  CIs,  and  MRs.  Force  the 
source  selection  team  to  concentrate  on  really  important 
areas. 

2LT  —  with  more  people,  you  tend  to  get  a  more 
objective  view  of  the  proposal.  One  drawback:  it's  difficult 
to  track  a  lot  of  people  to  ensure  they  complete  their 
evaluations  -  especially  when  the  source  selections  long 
(greater  than  3  months)  . 


ELECTRONIC  SYSTEMS  DIVISION 

GS-13  —  A  few,  qualified,  competent  individuals  can 
avoid  much  useless  debate  and  argumentation. 

GS-13  —  The  most  advantageous  choice  for  the  Air  Force 
can  usually  be  determined  quickly  and  with  a  minimal  number 
of  people. 

GM-13  —  Less  conflict  of  individual  people's 
understanding  of  Govt  requirements,  especially  as  relates  to 
software. 

CAPT  —  Less  people  involved  -  saves  time  and  easier  to 
establish  concensus  (if  desirable) . 

GS-13  —  It  would  make  the  evaluation  move  more 
quickly. 


GS-13  —  There  are  fewer  "editors"  and  the 
responsibility  for  a  given  area  can  be  established.  In 
addition,  the  single  evaluator  tend  to  be  more  concise. 

GS-13  —  If  they  are  top-notch,  qualified,  and 
dedicated  solely  to  the  source  selection. 

1LT  —  Avoids  redefining  or  further  defining  system 
requirements . 

1LT  —  Less  coordination,  more  dedication. 

GM-14  —  No  focus  on  large  panels. 

SES  —  Fewer  people  create  fewer  problems  (assumes 
minimum  necessary  has  been  satisfied) . 

GM-15  —  Fewer  opinions  to  meld  into  a  single  product. 

GS-12  —  Usually,  the  square  root  of  the  number  of 
persons  assigned  on  SSEB  perform  almost  all  the  analysis 
after  the  initial  evaluation. 

GS-13  —  Typically,  large  source  selections  produce  too 
much  unsupportable  "evaluations"  by  folks  who  don't 
understand  the  program  in  sufficient  detail,  or  are  not 
"expert"  in  their  field.  This  places  extra  burden  on 
item/area  managers  to  ensure  fairness  and  accuracy. 

CAPT  —  It  will  limit  the  paperwork  and  varying 
opinions. 

GS-13  —  Easier  coordination,  more  consistent 
evaluations,  hopefully  less  turnover  of  personnel. 


SPACE  DIVISION 

MAJOR  —  It  eliminates  functional  interests  that  are 
not  key  to  the  decision  process. 

LT  COL  —  Makes  it  politically  more  feasible  to 
eliminate  observers  from  misc  commands  outside  of  the 
product  division  who  have  little  to  contribute. 

GM-15  —  If  you  have  the  correct  disciplines 
represented . 

GM-13  —  Fewer  people  make  decision-making  possible. 
There  should  be  more  people  in  an  advisory  nature. 


COL  -  To  essential  minimum  number.  Speeds  process;  more 
inclined  to  focus  on  important  issues. 

GS-13  —  Fewer  people  would  result  in  greater  knowledge 
of  total  evaluations  by  each;  Better  understanding  of  the 
total  proposal. 

CAPT  —  Less  people  to  deal  with  in  reaching 
conclusions  or  recommendations  concerning  proposals. 

GM-13  —  Provided  experienced,  knowledgeable  people  can 
be  found.  5-10  people  can  perform  a  meaningful,  un-biased 
evaluation. 

GM-13  —  Ten  qualified  evaluators  are  as  good  as  a 
board  with  fifty  with  only  ten  really  qualified.  Significant 
processing  time  is  required  to  screen  excess  opinions. 

1LT  —  Eliminates  the  number  of  assessments  that  must 
be  consolidated,  also  cuts  down  on  the  number  of  review 
levels . 

GM-15  —  We  always  limit  number.  Simplifies  management 
of  source  selection. 

CAPT  —  Less  data  to  integrate.  Higher  probability  of 
concensus.  Risk  -  individual  biases  can  dominate. 


BALLISTIC  MISSILE  OFFICE 

MAJOR  —  Ensures  SSEB  chairman  plans  day  to  day 
activity  of  evaluation  and  promotes  quality  of  evaluations. 
When  limiting  number  of  evaluators,  it  is  essential  that  the 
personnel  be  chosen  well.  Gets  evaluation  accomplished  on 
the  best  schedule  attainable. 

GM-14  —  Cuts  time  and  number  of  meetings. 

GM-14  —  Saves  time  and  effort  involved  in 
consolidating  evaluations  and  prevents  fragmentation  of 
information. 

COL  —  Result  is  you  get  better  qualified  people  who 
can  devote  more  time  to  the  evaluation. 

LT  COL  —  Too  many  people  produce  a  consensus  report  - 
the  smaller  the  committee,  the  more  likely  that  divergent 
opinions  will  surface. 

COL  —  Again,  focuses  attention  on  the  most  important 
aspects. 


COL  —  Then  dialogue  can  take  place.  Also,  small 
proposals  allow  all  SSEB  members  to  read  the  whole 
proposal  -  avoids  out  of  context  evaluations. 

COL  —  If  proposals  are  page  limited,  a  small  number  of 
evaluators  can  read  each  proposal  and  prepare  comments. 
Consolidation  of  the  comments  into  strong  point/def iciency 
statements  is  less  time-consuming. 


ARMAMENT  DIVISION 


LT  COL  —  Consolidating  comments  of  multiple  evaluation 
is  time  consuming. 

GM-13  —  A  small  number  of  good  evaluators  is  faster 
and  more  meaningful  than  a  large  number  of  "semi-qualified" 
individuals  who  due  to  limited  knowledge  of  requirements 
slows  down  the  process. 

GM-15  —  Fewer  differences  to  reconcile. 

GM-14  —  Evaluation  results  can  be  more  easily  and 
quickly  assembled. 


EASTERN  SPACE  AND  MISSILE  CENTER 


GM-13  —  Utilize  the  best  technical  people  for  the 
SSEB.  Look  for  qualified  individuals  -  quality  in  lieu  of 
quantity. 

GM-13  —  Easier  to  come  to  a  resolution  with  a  lessor 
amount. 

GM-13  —  Prevents  paper  mills. 


AEROMEOICAL  SYSTEMS  DIVISION 

LT  COL  —  Better  exchange  of  info. 

LT  COL  —  Shorten  time  required  for  evaluation. 

MAJOR  —  The  larger  the  board,  the  slower  tnings  work! 
MAJOR  —  Accelerate  the  process. 


AIR  FORCE  AIR  LOGISTICS  CENTER 


GS-12  —  If  good  people  and  willing  to  give  evaluators 
credit  for  some  level  of  expertise. 


NO  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 

MAJOR  —  Relative  to  size  of  proposal  and  how  many. 

MAJOR  —  The  numbers  should  be  allowed  to  vary 
depending  on  complexity  and  number  of  respondees. 

GM-14  —  Other  than  $  spent  on  evaluation,  the  number 
of  people  has  little  effect.  The  number  of  functions 
evaluated  has  an  effect.  Many  organizations  feel  that  for 
them  to  be  able  to  hold  the  contractor's  feet  to  the  fire 
during  contract  execution,  they  must  evaluate  the 
contractors  proposals.  This  is  not  true.  Only  those  areas 
that  are  discriminators  need  reviewed  during  proposal 
evaluation.  I  do  not  recall  a  time  when  configuration 
management,  data  mgt,  LCC  mgt,  corrosion  control,  etc.  made 
a  difference  on  who  was  awarded  a  contract.  Yet  we  burn 
thousands  of  hours  of  critical  resources  (who  should  be 
monitoring  active  contracts)  in  the  source  selection  process 
evaluating  these  areas.  Suggest  source  selections  be  limited 
to  those  discriminators  such  as  technical  approach, 
schedule,  manufacturing  capability,  logistics 
considerations,  reliability  be  evaluated. 

GM-15  —  Most  tech  eval  people  are  functionally 
compar tmented  -  need  to  cover  all  aspects. 

MAJOR  —  Neither  advantage  nor  disadvantage. 

GM-14  —  The  number  of  people  on  the  33EB  has  no 
beacing  on  completing  the  evaluation  process  -  the  same 
effort/time  is  required. 

2LT  —  If  kept  within  reason,  the  more  inputs  available 
the  more  informed  the  33A  will  be. 

GM-14  —  The  process  is  complex  and  arduous.  Limiting 
the  number  of  evaluators  would  only  compound  the  problems. 

GS-12  —  Complex  programs  can  benefit  from  eval  by 
multiple  disciplines  or  multiple  opinions.  Limiting  might 
be  OK  on  smaller  programs  or  low  complexity  programs. 

LT  COL  —  Could  hamper  needed  experience,  requiring 
some  individuals  to  work  outside  primary  specialty. 
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MAJOR  —  Many  clarifications  to  requirements  can  be 
resolved  during  the  S3  process.  If  you  limit  people,  major 
issues  can  be  overlooked. 

GS-12  —  Different  people's  ideas  are  needed  so  that  an 
important  area  is  not  overlooked. 

GS-12  —  Each  contract  has  different  requirements-  one 
may  need  more  technical  analysis,  where  other  would  not. 

CAPT  —  Ideally,  there  should  be  one  person  to  evaluate 
each  factor.  Limiting  people  without  limiting  factors  or 
imposing  page  limits  would  result  in  missed  schedules. 

GM-13  —  In  general,  my  experience  shows  that  only  a 
small  number  of  individuals  are  required  full-time.  Others, 
such  as  advisors,  counselors,  can  be  called  on  as  required. 

GM-13  —  As  long  as  the  people  are  assigned  to  an  Area 
Chairman  who  has  control,  the  "extra"  evaluators  contribute 
to  a  good  thorough  review  by  people  who  are  experts  in  their 
field. 

CAPT  —  Each  program  size  and  scope  determines  the  size 
required  to  accurately  conduct  the  SS. 

Gs-14  —  It  is  always  hard  to  get  qualified 
evaluators  -  so  I  have  never  seen  too  many. 

GS-12  —  Generally,  not  enough  personnel  available  to 
begin  with.  The  SSs  I'm  familiar  with  have  never  requested 
personnel  that  were  to  essential  for  success. 


3LSCTR0NCC  SYSTEMS  DIVISION 

GM-15  —  Number  of  people  assigned  has  to  be  consistent 
with  the  complexity  of  the  procurement. 

2LT  —  It  takes  a  lot  of  work  and  would  pull  the  fewer 
people  off  their  jobs  longer. 

CM-15  --  Don't  have  enough  people  to  do  the  job  now. 
With  fewer  evaluators,  will  take  more  time,  overwork  staff, 
get  sloppy  results.  Most  SSEB  members  are  totally 
inexper ienced . 

MAJOR  —  Depends  upon  the  program. 

GS-12  —  It  depends  on  the  complexity  of  the 
anticipated  acquisition.  A  standard  limit  may  be  seriously 


harm  the  effort  if  insufficient  resources  are  available  to 
evaluate  the  proposals. 


GS-12  —  Need  all  the  technical  expertise  you  can  get. 

CAPT  —  It  is  hard  to  assemble  an  experienced  group  of 
evaluators.  Usually,  the  experienced  individual  is 
designated  as  area  chief.  In  this  situation,  the  flexibility 
to  assemble  your  team  on  knowledge  instead  of  number  of 
people  benefits  the  government. 

GM-14  —  Depends  on  the  program  and  most  SSEBs  aren't 
overstaffed . 

GM-13  —  A  complete/thorough  review/evaluation  cannot 
be  performed. 


SPACE  DIVISION 

COL  —  Use  as  many  people  as  are  necessary  to  get  the 
job  done. 

CAPT  —  Dependent  on  expected  response/detail. 

GS-13  —  More  brains  should  result  in  a  better  product. 

GS-13  —  The  limits  would  be  overcome  by  the  use  of 
advisors. 

GS-14  —  Artificial  limits  are  impractical.  All 
necessary  people  should  be  used. 


ARMAMENT  DIVISION 

GM-15  —  Fails  to  consider  unique  aspects  of  each 
program. 

GM-13  —  Should  have  adequate  depth  to  thoroughly 
evaluate  proposals. 

GM-15  —  We  only  use  what  we  need. 

GS-12  —  The  problem  is  not  the  number  of  people  but 
the  quality  of  people. 


AIR  FORCE  AIR  LOGISTICS  CENTER 

GS-12  —  Because  evaluators  are  not  typically  seasoned 
veterans  with  at  least  10  years  experience;  nor  are  item 
captains  or  area  chairraenl 


SURVEY  QUESTION 

K.  Would  a  pre-established  time  limit  between  proposal 
receipt  and  contract  award  be  an  advantage  to  the  proposal 
evaluation  process? 

YES  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 

GM-14  —  We  normally  take  the  time  we  are  allowed  but 
longer  time  doesn't  improve  the  evaluation  quality. 

LT  COL  —  Help  keep  matrixed  functional  support  on 
schedule. 

GM-13  —  Work  wouldn't  be  generated  to  fit  the  time 
available  -  shorten  turnaround  to  get  workers  back  on  job. 

MAJOR  —  A  defined  amount  of  time  for  the  evaluation 
would  allow  individuals  (PMs)  more  flexibility  in 
assigning/completing  work. 

GM-13  —  Work  seems  to  expand  to  fill  allotted  time. 
Lack  of  a  time  limit  causes  the  process  to  slip  out  further 
and  further. 

GM-15  —  Confine  effort  to  a  prescribed  time  again 
would  imply  acceptance  of  a  reduced  quality  product. 

GM-14  —  It  keeps  evaluators  focuses  on  the  task  at 

hand . 

MAJOR  —  I  thought  we  do  this  anyway.  Establish 
schedules  in  planning  phase  consistent  with  requirements  and 
resources  available. 

GS-13  —  But  not  a  good  idea  because  "contract  award 
issues"  could  cause  delay.  Contract  awards  should  be  driven 
by  "need". 

GM-15  —  As  a  goal  -  yes.  However,  certain  unplanned 
circumstances  arise  which  must  be  dealt  with  such  as  buy- 
ins,  all  costs  beyond  program  budget,  and  no  adequate 
technical  approach  proposal. 

LT  COL  —  Get  it  donel 

2LT  —  Reduces  the  chance  of  a  SS  languishing  and 
bogging  down  in  the  evaluation. 


GS-13  —  Establishes  working  goals  -  useful  for  any 

task. 

MAJOR  —  To  some  extent.  Flexibility  would  be 
appropriate  in  this  area. 

GS-12  —  Limit  the  nit-picking  CRs  and  DRs. 

GS-12  —  Procrastination  can  always  aelay  contract 
award. 

CAPT  —  Because  either  people  would  work  24  hours 
day/  7  days  a  week  or  they  would  tailor  factors  and  items, 
impose  page  limits,  and  take  other  steps  to  shorten  the 
selection  process.  The  quality  of  a  selection  is  uneffected 
whether  it  takes  2  months  or  6  months.  The  amount  of 
paperwork  generated  is  the  only  difference. 

GM-14  —  Would  force  all  aspects  of  the  evaluation  to 
truly  complete  the  evaluation  in  a  specified  time  period. 
Currently,  unless  there  is  a  truly  strong  manager  in  charge 
of  the  process,  the  evaluation  is  permitted  to  continue 
almost  unconstrained  by  schedule. 

GM-13  —  Set  a  closure  date  to  end  the  questions  and 
get  on  with  the  job  of  awarding  a  contract.  Also,  the  set 
time  should  be  a  part  of  the  overall  program  schedule  to 
have  an  efficient  utilization  of  funds  and  resources. 


ELECTRONIC  SYSTEMS  DIVISION 

GS-13  —  only  if  tailored  for  each  source  selection. 
Only  if  the  procedures  used  can  be  accomplished  in  that  time 
limit.  Program  complexity,  number  of  proposals,  and  size  of 
evaluation  team  strongly  influence  the  amount  of  time 
required. 

GS-13  —  If  the  time  limit  is  reasonable,  it  would 
identify  specific  milestones  to  measure  progress. 

1LT  —  Avoids  technical  leveling.  The  longer  the  source 
selection  process  goes  on,  the  greater  the  chance  that  the 
Government  will  inadvertantly  give  "clues"  to  the  deficient 
offerors  to  bring  them  up  to  par  with  the  others. 

1LT  —  Defines  schedule  apriori. 

SES  —  Work  tends  to  expand  to  fill  the  time  available. 

CAPT  —  Schedules  during  source  selection  are  hard  to 
keep.  We  had  situations  where  the  evaluation  results  were 


briefed  to  ESO/CC  and  then  he  assembled  another  team  to 
verify  the  results.  Even  after  we  had  results  approved  by 
SSAC. 


GS-13  —  If  the  time  limit  was  obtainable,  it  would 
result  in  quicker  contract  award.  Some  source  selections 
seem  to  be  dragged  out  with  several  slow  periods. 

GM-14  —  Too  many  source  selections  are  extended  and 
tie  up  personnel  unproductively .  Making  a  decision  does  not 
always  require  full  detail  of  circumstances  and  is  seldom 
provided  to  the  SSA. 

GS-12  —  Goals  are  set  to  be  met. 

CAPT  —  If  reasonable,  forces  decisions  (not  more  and 
more  questions).  Speeds  an  already  very  long,  laborious 
process. 

2LT  —  It  would  probably  cause  the  RFP  to  require  more 
concise  proposals  and  thus  reduce  the  time  between  proposal 
receipt  and  contract  award. 


SPACE  DIVISION 

LT  COL  —  Prevents  other  priorities  from  interfering 
I  with  the  process. 

1LT  —  Help  speed  staff  review  and  coordination. 

CAPT  —  Forces  a  structure  on  source  selection. 

I  GM-13  —  We  do  this  to  confine  the  time  of  the 

evaluation  process. 

GS-12  —  Time  might  be  spent  more  efficiently  if  there 
existed  a  time  limit,  although,  attention  must  be  given  to 
the  realism  of  the  time  limit. 


BALLISTIC  MISSILE  OFFICE 

MAJOR  —  Forces  good  planning,  best  utilization  of 
personnel,  and  focuses  the  evaluation  on  key  discriminators 
that  should  be  considered  by  the  SSA  in  arriving  at  a 
decision. 

COL  —  It  enforces  a  discipline  on  a  bureaucratic 
process. 
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COL  —  It  allows  for  advance  planning  by  govt,  and 
contractors  to  make  resources  available  at  the  right  time. 

GM-14  —  Structures  the  evaluation. 

GM-14  —  Only  if  commander  enforces  limit  -  people  take 
as  much  time  to  evaluate  as  they  are  given. 

COL  —  It  helps  establish  priorities  with  other 
projects  competing  for  time  of  all  people  involved  in 
evaluation  and  contract  production. 

LT  COL  —  It  encourages  decisions  rather  than  delay. 

GM-13  —  Source  selection  will  expand  to  fill  the  time 
available.  A  pre-established  limit  (with  some  SSA 
flexibility  for  exceptions)  is  useful  in  forcing 
participants  to  plan  efficiently. 


ARMAMENT  DIVISION 

GM-13  —  People  take  advantage  of  procrastinating  when 
no  schedule  is  set. 

GM-14  —  Would  require  process  to  be  completed  on  a 
schedule. 

GM-13  —  Depends  on  the  complexity  of  program  under 
evaluation. 

GS-12  —  There  already  is  a  time  limit  established,  at 
least  here  at  AD  there  is.  Schedules  slip,  of  course,  but  we 
try  to  meet  the  schedules. 
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GM-13  —  It  would  define  the  time  available. 


AEROMEDICAL  SYSTEMS  DIVISION 

LT  COL  —  Force  contracting  to  keep  moving. 

LT  COL  —  Forces  the  system  to  be  responsive. 

MAJOR  —  Murphy's  Law  -  What  ever  the  time  allowed,  the 
work  will  expand  to  fill  it. 

MAJOR  —  Eliminate  bureaucratic  "CYA"  delays. 


AIR  FORCE  AIR  LOGISTICS  CENTER 


GS-12  —  It  would  force  managers  to  weed  out 
unnecessary  actions;  also  contract  types  must  streamline 
their  processes.  A  formal,  mid-term  review  seems  superfluous 
since  the  data  upon  which  it  is  based  (the  as-received 
proposals)  always  gets  modified  by  CIs  and  ORs  and  sometimes 
MRs. 


NO  RESPONSES 


AERONAUTICAL  SYSTEMS  DIVISION 

GM-13  —  These  limits  would  end  up  driving  the  system 
when  each  source  selection  is  different. 

LT  COL  —  Enough  pressure  on  the  schedule/people  now. 
Artificial  constraints  would  only  add  to  an  already  "nervous 
tummy" . 


CAPT  —  Many  problems/issues  are  elevated  due  to  the  SS 
process. 

GS-14  —  Too  much  pressure  on  the  SSEB.  Quality  is 
better  than  quantity.  Also,  most  evaluators  have  other  jobs 
and  must  devote  some  time  to  them. 

GS-12  —  Each  source  selection  usually  has  it  unique 
problems  which  don't  permit  award  on  schedule.  The  SSP 
already  has  a  schedule  which  is  the  same  as  suggested  above. 

GM-15  —  We  have  schedules  now  which  we  seldom  meet. 

GS-12  —  Every  source  selection  that  I  have  been  a  part 
of  has  had  a  tight  proposal  evaluation  schedule.  After  the 
decision  gets  past  the  SSEB  is  when  the  slow  down  happens. 

GM-14  —  It  would  just  be  broken.  It  takes  what  it 
takes  due  to  the  problems  at  hand.  The  tracking  would  just 
be  another  source  of  SSEC  problems.  Nobody  likes  to  remain 
in  SS  very  long,  so  incentive  is  always  to  get  out. 

GM-14  —  However,  I  believe  a  good  tight  schedule 
tailored  to  each  program  is  needed.  Time  can  be  reduced. 

Some  tasks  can  be  done  in  parallel. 

GM-15  —  Major  decisions  will  always  be  made  by  people 
too  important  to  place  rigid  deadlines  on.  Might  be  OK  for 
small  programs. 


MAJOR  —  Evaluation  of  proposal  not  the  problem  -  tend 
to  further  define  criteria  and  the  decision  process  eats  up 
the  time  -  proposal  evaluation  is  not  the  big  time  eater. 

GM-14  —  You  already  have  a  schedule  that  you  are 
working  to  -  the  problem  is  in  the  number  of  CRs  and  DRs 
that  evolve  and  must  be  resolved. 

MAJOR  —  Each  solicitation  is  unique,  to  impose  a  time 
limit  would  be  detrimental  to  all  source  selections. 

LT  COL  —  Govt  deadlines  tend  to  result  in  rushed,  and 
less  than  complete  effort.  (However,  schedules  are 
important;  job  must  be  completed  ASAP) . 

GM-13  —  People  tend  to  take  as  long  as  they  have  to 
complete  evaluation. 

GM-13  —  It's  already  done  now,  but  if  a  problem 
develops,  the  extra  time  will  always  be  taken  to  resolve  it. 

GS-12  —  Complexities  within  the  requirements  -  a  small 
business  contractor  would  require  less  time  to  evaluate. 

2LT  —  It  would  just  force  schedule  slips. 

GM-13  —  When  a  time  schedule  is  established,  it  is 
usually  too  short  to  do  a  good  evaluation  effort.  If  a 
reasonable  schedule  was  established,  I  would  agree.  The 
problem  arises  because  the  schedule  is  established  prior  to 
receipt  of  proposals,  and  hence  the  complexity  of  the 
evaluation  is  not  known  i.e.  number  of  offerors,  etc. 


ELECTRONIC  SYSTEMS  DIVISION 

GS-13  —  The  source  selection  process  requires 
flexibility.  A  time  limit  is  not  an  adequate  substitute  for 
good  management. 

GS-13  —  Time  limits  tend  to  be  unrealistic.  A  source 
selection  tends  to  have  a  "time  sense"  of  it  own  factors 
tend  to  lengthen  source  selections,  irrespective  of 
established  schedules. 

GS-13  —  Arbitrary  deadlines  result  only  in  rushed, 
incomplete  tech  evals,  which  makes  it  more  difficult  to 
develop  legitimate  discriminators  upon  which  to  base  a 
source  selection  decision.  Also,  the  biggest  variable  in 
time  is  the  number  of  proposals  received  in  response  to  RFP, 
which  you  don't  know  with  certainty  until  receipt  date. 
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GM-14  —  Artificial  constraint. 

GM-15  —  Impractical  -  cant  foresee  all  circumstances. 

GS-12  —  Of  the  five  source  selections  I  have 
participated  in,  no  two  were  alike  except  for  format,  each 
encountered  different  issues  that  needed  varying  periods  of 
time  to  resolve. 

CAPT  —  Because  of  the  funds,  this  may  change  and  you 
will  have  to  hold  up  source  selection. 

GS-13  —  The  time  required  is  strongly  dependent  on  the 
number  of  proposals  received. 

MAJOR  —  Some  proposals  may  not  lend  itself  to  a 
definite  time  limit. 

GS-12  —  The  added  pressure  of  mandated  time 
constraints  would  only  increase  the  stress  levels  of  this 
"dreaded"  activity.  People  are  already  attempting  to  do 
their  best  under  difficult  circumstances  -  and  depending  on 
the  technology,  etc.,  the  mission  of  the  program  could  be 
seriously  impacted. 

GS-12  —  Forcing  evaluators  to  rush  could  only 
jeopardize  a  source  selection. 

GM-14  —  Schedule  is  by-product,  not  an  end.  Technical 
understanding,  fair  and  reasonable  cost  determination  are 
the  real  needs,  and  the  time  necessary  to  achieve  them  will 
vary  with  each  program. 

LT  COL  —  The  time  limit  should  already  be  in  the 
source  selection  plan  anyway.  Arbitrary  limits  never  work. 
Limits  must  be  set  based  on  size  of  effort. 

GS-13  —  Time  for  the  source  selection  is  greatly 
dependent  on  the  number  of  proposals  received.  If  the  time 
limit  was  too  restrictive,  it  could  cause  erroneous 
decisions. 

GM-15  —  It's  hard  to  establish  firm  time  limits  as 
there  are  so  many  exceptions  to  the  rule. 

GM-15  —  Absurbl  Don't  tie  Govt's  hands  -  whats 
different  here  than  just  having  an  internal  schedule  for 
management  purposes? 


SPACE  DIVISION 


MAJOR  —  A  source  selection  decision  can  not  be  an  end 
unto  itself.  Time  limits  could  force  decision  before  all 
factors  are  known  and  price  negotiated. 

GM-15  —  Too  restrictive. 

GM-13  —  Would  only  create  more  reporting  practices. 
Less  flexibility  and  a  possible  disservice  to  the  Air  Force. 
Sometimes  it  is  better  to  take  a  little  longer  and  be  sure 
the  best  approach  has  been  selected. 

COL  —  Are  too  many  factors  to  be  considered  to  be  too 
arbitrary  on  govt  side  of  the  equation.  Time  goals  are  OK  so 
long  as  they  are  not  absolutes. 

GS-13  —  Little  benefit  and  possibly  of  catastrophic 
consequences. 

GM-13  —  There  are  currently  guidelines  and  there 
already  exists  too  many  government  "pre-established"  limits 
which  make  good  management  nearly  impossible. 

GM-13  —  Every  procurement  is  unique.  Guidelines  -  yes; 
Limits  -  no. 

GM-15  —  Too  many  unknowns,  we  have  found  it  better  to 
slip  schedule  and  do  a  good  evaluation. 

COL  —  Use  whatever  time  is  necessary  to  get  the  job 
done  correctly  and  thoroughly. 

CAPT  —  Soft  no.  A  time  limit  would  constrain 
evaluation  quality  given  limited  SSEB  resources  often  found 
in  source  selections.  However,  without  a  time  limit,  a  fair 
and  reasonable  response  cannot  be  assured. 

GS-13  —  It  would  put  undue  pressure  on  the  evaluators. 
A  quality  job  might  not  be  produced. 

GS-13  —  Force  time  limits  would  increase  the 
probability  of  poor  procurements. 

LT  COL  —  It  is  impossible  to  insure  compliance  with  a 
mandatory  schedule,  and  almost  all  source  selections  will 
have  a  schedule  of  some  kind,  including  a  desired  award 
date.  So,  time  limit  is  already  pre-established. 

CAPT  —  It  is  more  important  to  have  a  thorough  and 
fair  evaluation  than  a  very  quick  one. 

GS-14  —  Artificial  limits  are  impractical. 
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ARMAMENT  DIVISION 


LT  COL  —  Would  be  too  many  exceptions  where  it  was  to 
the  government's  advantage  to  extend  the  source  selection. 

GM-15  —  If  time  is  a  problem,  it's  caused  by  poor 
organization  or  direction. 

GM-15  —  Because  time  limits  are  based  on  what  is 
wanted  rather  than  what  is  needed  1 

GM-13  —  People  in  a  hurry  make  mistakes  -  you'll  pay 
later . 

GS-14  —  You  don't  know  what  it  should  be  to  assure 
source  selection  objectives  are  achieved.  It  could  even 
unnecessarily  induce  prolongation.  GAO  would  not  be 
impressed  by  a  time  limit  in  assessing  the  propriety  of 
source  selection  conduct. 


EASTERN  SPACE  AND  MISSILE  CENTER 

GM-13  —  Depending  upon  the  complexity  of  the 
acquisition  and  number  of  offerors  dictate  time  required  to 
evaluate  and  award. 

GM-13  —  Complexities  may  overweigh  time 
considerations. 


AEROMEDICAL  SYSTEMS  DIVISION 


GS-12  —  I  don't  believe  hasty  decisions  would  be  to 
our  advantage.  In  my  experience,  the  evaluation  team  is 
usually  short  on  manpower  and  has  very  limited  experience.  A 
lot  of  money  and  resources  is  riding  on  their  decision  and 
it  should  not  be  rushed. 


PART  III  Other  Comments 


AERONAUTICAL  SYSTEMS  DIVISION 

GM-13  —  The  single  biggest  factor  to  BAD  source 
selections  is  because  of  a  rush  job  to  put  the  program  on 
contract  quickly  and  later  we  suffer  the  penalty  of 
correcting  the  deficiencies  which  take  more  time  than  early 
planning  would  require  and  spending  money  we  can  ill  afford 
to  waste. 

GS-12  —  The  ASD  source  selection  facility  is  in  dire 
need  of  expansion.  It  lacks  anywhere  near  the  spare  media  to 
conduct  our  SSs.  The  facility  should  not  be  limited  to  AFR 
70-15  SSs.  Suggest  more  formal  SS  training. 

GS-14  —  I  think  for  small  $  contracts,  we  do  need  a 
quicker  way  to  award  contracts.  Too  many  regulations  impede 
timely  awards.  Even  competing  small  $  contracts  (i.e.  < 
$100,000)  is  very  expensive  and  dilutes  government  resources 
for  the  larger  ones  that  really  count. 

GM-15  —  The  requirement  for  a  flawless  product,  (No 
ASD  source  selection  protest  has  ever  been  sustained) 
demands  large  numbers  of  people  and  long  periods  of  time  but 
we  do  get  the  best  deal  for  the  US  government  and  treat  each 
bidder  fairly. 

GM-13  —  The  current  source  selection  process  is: 

1.  A  waste  of  the  tax  payers  dollars- 

a.  tooo  many  employees  involved. 

b.  too  many  briefings  to  go  through 

c.  personalities  enter  into  decisions. 

An  alternative  approach  to  the  current  system: 

1.  Time  limit  based  on  some  pre-established  criteria 
(dollars,  type  of  requirement) 

2.  DO  NOT  IDENTIFY  CONTRACTOR  BY  OTHER  THAN  ALPHA  OR 
NUMERICAL  INDICATOR  TILL  SELECTION  (ELIMINATES  PRE- 
ESTABLISHED  CHOICE) . 

3.  Base  decision  on  proposal,  no  limit  on  pages  and  no  CR 
or  DRs . 

4.  use  recommendation  by  the  CAS  or  PRO'S  on  pre-award 
surveys.  Don't  send  out  a  full  blown  ASD  to  perform. 

GM-15  —  The  key  to  improving/simplifying  the  source 
selection  process  is  maximum  delegation  of  authority  and 
responsibility  (which  we  don't  do  well  within  the  govt).  We 
do  a  good  job  of  selection  on  large  acquisitions  which  use 
the  conventional  approach.  Its  slow,  consumes  many  manhours, 
etc.,  but  its  generally  accurate  and  "controversy"  proof. 
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It's  with  the  large  number  that  would  be  more  appropriately 
selected  under  less  stringent  procedures,  that  we  have  the 
problem;  i.e.  we  overkill  the  small  stuff.  We  also  need  to 
do  a  better  job  on  scrubbing  data  that  we  ask  for  -  appears 
to  be  a  lot  of  "just  in  case"  stuff  that  we  don't  need. 

GM-14  —  If  you  are  attempting  to  get  a  standard  for 
all  source  selections,  that  is  a  mistake.  Each  evaluation 
should  be  judged  on  its  own  goal.  All  SS  processes  have  a 
proper  time  for  application.  Factors  to  be  considered  is  1) 
risk  to  government;  2)  time;  3)  cost;  4)  technical  or 
management  challenge. 

MAJOR  —  Although  I  have  only  helped  in  a  SS,  I  have 
reaped  the  problems  of  several  SSs.  The  low  bidder  won  -  but 
was  unable  to  manage  the  program.  I  would  stress  past 
performance  and  encourage  ASD  managers  and  PCOs  to  initiate 
contractor  performance  files  for  both  positive  and  negative 
actions . 

GM-14  —  Need  a  good  training  session  in  a  classroom 
atmosphere . 

GM-15  —  The  source  selection  process  is  a  painful  one, 
but  on  large  programs  it  probably  should  be. 

3ES  —  Part  II  misses  an  important  ingredient  in  the 
selection  of  the  approach.  The  type  of  program  is  usually 
the  driving  factor  in  selecting  an  approach.  Source 
selections  are  conducted  on  laboratory  technology  programs. 
Additionally,  conceptual,  demonstration/validation,  and 
full-scale  development  phases  are  competed  by  the 
acquisition  offices  in  ASD. 

Acquisition  strategy  also  plays  a  part  in  the 
selection  of  an  approach.  Many  times,  multiple  contractors 
are  selected  to  continue  work  in  a  parallel  effort  until  a 
follow-on  source  selection  is  made. 

ASD  often  times  uses  the  conventional  approach 
but  limits  the  proposal  page  count. 

The  complexity  of  the  proposal  evaluation 
standards  influences  the  number  of  people  needed  and  the 
time  duration  to  evaluate  a  proposal. 

GS-12  —  My  opinion  is  that  most  source  selections  are 
too  long.  In  1970  on  the  SCAD  program,  we  evaluated  18 
proposals  and  awarded  6  contracts  within  a  12-13  week 
period.  I  also  evaluated  5  or  6  revisions  of  the  Carrier 
(Boeing)  Aircraft  contractors  proposal  during  the  same  time 
frame.  Our  SPO  prepared  the  performance  specifications.  I 
would  like  to  consider  this  as  a  goal  for  source  selections 
of  a  comparable  weapon  systems  contractors.  Simulators  can 
be  done  in  a-  similar  time  frame  if  we  don't  try  to  level 


each  offeror.  Limit  the  source  selection  process  to 
experienced  evaluators  (trainees  would  be  observers  until 
proficiency  is  demonstrated) . 

GS-12  —  A  training  requirement  is  a  must.  Every 
person  used  as  an  evaluator  should  be  required  to  attend  a 
comprehensive  course  on  source  selection. 

GM-15  —  The  questionnaire  nibbles  at  the  crux  issues 
of  source  selection  without  getting  to  the  meat. 
Questionnaires  have  been  used  before  without  much  success  — 
see  PM  for  recent  more  detailed  questionnaire  ar 1  their 
answers.  One  good  man  who  understands  the  process  is  better 
than  all  questionnaires. 

CAPT  —  I  reiterate,  source  selections  are  a  waste  of 
government  resources.  The  source  selection  acts  a  a 
justification  for  what  aver  choice  the  3SA  makes  independent 
of  the  results  of  the  SSEB  evaluation.  Any  steps  which 
mandate  a  simpler  shorter  process  are  steps  well  taken.  AFR 
70-15  is  the  place  to  start. 


ELECTRONIC  SYSTEMS  DIVISION 

GS-13  —  I  believe  much  can  be  done  to  simplify  and 
reduce  the  time  to  select  a  "winning"  contractor.  I  have 
many  ideas  and  suggestions  to  do  so.  The  current  climate  at 
ESD ,  however,  is  not  receptive  to  any  "unconventional" 
approaches.  Some  of  the  most  productive  areas  that  could  be 
improved  are  listed  as  follows: 

1.  Improved  standard  evaluation/rating  forms. 

2.  Reduced  role  of  SSAC  (even  eliminated). 

3.  Better  training  of  SSEB  members. 

4.  Dedicated  resources;  i.e.  secretaries,  computers, 
facilities,  SSEOs,  etc  to  support  source  selections. 

5.  Allowing  known  experts  (non-govt)  to  evaluate,  rate, 
and  rank. 

6.  Recognition  that  selecting  the  optimum  offer  is 
relatively  simple  but  definitizing  a  fair,  comprehensive, 
contract  (often  fixed  price)  is  the  most  difficult,  time 
consuming  part  of  source  selection. 

7.  Reverse  current  trend  for  ESD  commander  to  be  SSA  on 
even  small  dollar  acquisitions. 

8.  Have  a  computerized  file  containing  detailed  past 
performance  information  on  the  most  common  defense 
contractors  eliminating  the  need  for  each  SSEB  to  create 
their  own. 

GS-13  —  For  conventional,  cost  team  should  have 
technical  advisor.  Funding  profiles  for  programs  in  source 
selection  should  be  frozen  to  prevent  funding  cuts  which 
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could  delay  the  source  selection  process.  A  source 
selection  fund  which  programs  with  small  funding  shortfalls 
could  draw  on  could  help  in  the  costs  where  proposals  are 
higher  than  funds  available,  the  number  of  areas,  and 
factors  should  be  minimized  to  limit  the  amount  of  paper 
work.  The  number  of  cost  formats  should  be  reduced.  A  lot 
of  time  and  effort  is  taken  up  by  the  offerors  responding  to 
our  ridiculous  requests  for  cost  data.  Computerized  tools 
would  be  helpful  to  allow  the  cost  team  to  manipulate  the 
cost  data. 

GM-14  —  4-step  is  not  suitable  for  development 
programs. 

GS-12  —  Courses  should  be  offered  on  how  to  fill  in 
CR's,  DR's,  PFN's,  Area  and  Item  summaries,  etc.  Some  S.S. 
elections  require  people  to  fill  in  and  refill-in  these 
forms  over  and  over  again  wasting  valuable  time.  Also  it 
needs  to  be  taught  what  constitutes  a  CR  vs.  a  DR;  when 
should  a  PFN  be  written;  what  types  of  questions  and  how 
should  you  ask  them,  in  face-to-face  negotiations?  Although 
the  SSAC  is  supposed  to  write  the  PAR  report  the  SSEB  does 
the  job-  how  do  you  write  one  of  those? 

How  do  you  prepare  a  GEMPC?  How  do  you  estimate  how 
long  it  will  take  a  contractor  to  do  a  job,  how  many  people 
will  be  involved,  what  is  the  level  of  effort?  These  things 
need  to  be  taught.  Many  evaluators  have  only  worked  for  the 
government  and  have  no  idea  how  long  it  takes  to  build  a 
plane  or  a  radar  or  how  many  people/or  months  it  takes  to 
get  ready  for  PDR;  or  how  many  secretaries  it  takes  to 
prepare  data  items.  The  GEMPC  is  the  most  guessed  at  and 
most  important  factor  in  a  source  selection. 

GS-12  —  Participation  in  the  source  selection 
experience  provides  valuable  insight  into  the  program  and 
the  acquisition  system  itself.  It  is  a  synergistic 
environment.  However,  source  selection  is  traditionally 
viewed  as  "something  wicked  this  way  comes".  (I’m  sure  Ray 
Bradbury  will  forgive  me  using  this  quote.)  People,  without 
experience  in  the  process,  suddenly  no  idea  as  to  how  to  go 
about  doing  what  they've  been  told  they  had  to  do. 

There  are  no  formal  training  courses  available  to  my 
knowledge  -  at  most  they  may  get  a  one  hour  briefing  on  the 
shalls  and  shall  nots.  The  courses,  at  least  the  ones  I 
have  attended,  have  spent  a  great  deal  of  time  on  how  to  put 
together  a  requirements  package.  What  you  learn  is  how  to 
state  the  tasking  and  requirements  -  what  language  to  use 
and  when  to  use  it.  No  course  provides  any  insight  as  to 
how  you  are  to  evaluate  an  offeror's  response  to  your 
package.  An  inexperienced  evaluator  might  have  ideas  as  to 
how  they  expect  the  offerors  to  respond  and  if  their  ideas 
are  not  reflected  in  the  proposal  its  time  to  hit  the  panic 
button. 


The  purpose  of  source  selection  has  been  mislaid  along 
the  way  (at  least  the  way  I  see  it) .  Evaluators  get  bogged 
down  on  evaluating  the  actual  design  -  something  that  must 
be  reserved  for  the  formal  reviews  after  contract  award  - 
and  forget  they  are  there  to  determine  whether  or  not  the 
proposals  reflect  an  adequate  approach  to  solve  the  problem. 
(I  realize  there  are  situations  where  the  source  selection 
purpose  is  to  chose  a  design.) 

As  far  as  establishing  mandated  time  constraints  -  if 
realistic  target  dates  are  scheduled  and  sufficient 
resources  provided  this  would  not  be  necessary.  Too  often, 
the  schedule  is  set  up  when  the  RFP  is  released  and  is  not 
revised  to  reflect  changes  (for  example,  if  an  offeror 
requests  and  is  granted  an  extension  to  the  proposal 
delivery  date) .  The  other  problem  is  the  annual  end-of- 
fiscal-year  crunch.  The  emphasis  shifts  from  getting  a  good 
contract  awarded  to  just  getting  the  flipping  contract 
awarded  so  the  money  isn't  lost. 

Mandating  the  approach  to  use  for  the  source  selection 
process  is  another  sore  spot.  It  becomes  a  struggle  between 
doing  what's  best  for  the  mission  and  staying  within  the 
regulations.  I  realize  that  there  are  laws  that  must  be 
adhered  to  but  there  is  a  definite  gap  between  staying 
"legal"  and  staying  within  the  regulations. 

Another  problem  that  I  have  noted  recently  is  a 
tendency  of  higher  headquarters  to  interfere  (malting 
suggestions  that  result  in  a  change  to  the  requirements  and 
bouncing  the  funding  profiles  around,  etc).  They've  given 
the  evaluators  a  job  to  do  -  let  them  do  it.  Yes, 
headquarters  has  a  vested  interest  but  the  role  should  be 
limited.  Their  role  is  usually  as  members  of  they  SSAC.  I 
suggest  making  them  active  members  of  the  board  -  that  way 
they  would  know  and  not  have  to  question  the  whys  of  the 
ratings.  It  would  also  save  time  during  the  process. 
Convening  the  SSAC  would  not  have  to  wait  for  the 
"convenience"  of  all  the  members.  An  evaluator's  time  is 
just  as  valuable  as  the  time  of  the  SSAC  Chairperson.  (It 
might  only  save  one  week  or  it  could  save  a  month  but  it 
would  still  be  time  saved.) 

Had  enough  of  the  problems?  How  about  some 
suggestions? 

Implement  a  formal  training  course  on  how  to  go  about 
evaluating  a  proposal  and  make  it  available  to  all  the 
people  involved  in  source  selection.  Perhaps  a  generic 
course  at  AFIT  followed  by  a  Product  Division  unique. 

Establish  a  course  on  pricing.  It's  a  nightmare  for  an 
experienced  data  manager  to  evaluate  quotes  for  data  costs. 
Imagine  the  terror  an  inexperienced  person  feels  when 
confronted  with  that  task. 

Include  in  the  pricing  course  a  how-to  segment  on 
developing  the  Government's  Estimate  of  Most  Probable  Cost 
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interpretation  of  the  resoutces  requited  for  each  offeror  to 
do  the  job  they  think  we  want  done  in  terras  of  labor  hours. 
How  can  you  reasonably  expect  an  inexperienced  person  to  do 
this?  And  if  you  have  more  than  one  offeror . 

Establish  a  core  group  (cadre)  of  experienced  personnel 
(volunteers  would  be  best)  to  provide  expertise  to  source 
selections.  Make  it  their  only  job.  Provide  a  back  up  for 
them  at  their  regular  assignment.  And  don't  have  the  same 
people  doing  all  the  selections  -  a  rotational  assignment 
list  may  work. 

Assign  inexperienced  personnel  as  "trainee"  SSEB 
members.  Let  them  go  through  the  whole  process  at  least 
once  under  supervision  before  they  get  assigned  as  item 
captains  or  area  chiefs. 

Let  the  evaluation  approach  be  flexible.  A  combination 
of  ASD,  BMO ,  and  conventional  may  be  best  in  one  case 
whereas  4  step  might  be  the  only  way  in  another.  Having  to 
justify  the  approach  takes  time  and  creates  frustration. 

Leave  the  money  alone  and  don't  change  requirements 
(unless  it  makes  sense  to  do  so)  . 

Eliminate  some  of  the  cost  formats  that  serve  no  real, 
useful  purpose  and  make  the  remaining  formats  easier  to 
understand. 

Have  experienced  people  write  the  IFPPs  with  the 
stringent  parameters  as  if  it  were  a  specification  or 
statement  of  work.  Confusion  reigns  supreme  if  the  IEPP 
contains  conflicts  or  areas  subject  to  individual 
interpretation. 

GS-13  —  The  major  problems  I  have  seen  in  source 
selections 

Lack  of  people  with  source  selection  experience, 
particularly  when  the  source  selection  is  designed  and  the 
evaluation  standards  are  written. 

Trying  to  evaluate  too  much.  There  is  a  tendency  to 
include  in  the  evaluation  things  that  cannot  affect  the 
outcome. 

Turnover  in  personnel  during  source  selection. 

Morale.  The  overtime,  schedule  pressures,  nickpicks, 
and  poor  facilities  all  contribute  to  poor  morale  and  a  very 
strong  desire  never  to  do  another  source  selection. 

GS-13  —  I  am  only  marginly  familiar  with  the  BMO  and 
ASD  approaches.  Prom  my  understanding  of  the  approaches,  in 
order  for  them  to  function  properly,  a  major  commitment  must 
be  made  by  the  commander  and  others  in  the  source  selection 
stream  to  keep  "hands  off"  during  the  process.  Since  ESD 
has  an  active  commander  and  other  "active"  staffs  this  my  be 
difficult  to  achieve. 

One  additional  observation;  Time  in  a  source  selection, 
if  made  too  short,  tends  to  cause  poor  decisions  and  much 
remedial  work  after  contract  award. 
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1LT  —  I  think  it  would  be  most  efficient  to  have  a 
permanent  Source  Selection  Team.  To  help  this  team  in 
evaluating  the  proposals  the  personnel  from  the  program 
office  would  brief  this  team  and  also  help  evaluate  the 
proposals.  This  would  be  especially  effective  in  the  cost 
area  as  most  evaluators  really  do  not  know  how  much  is  too 
much  or  too  little  for  given  tasks. 

Stop  selecting  the  lowest  bidder.  There  are  usually 
very  good  reasons  why  one  technical  approach  may  cost  a 
little  more  (  less  ECPs,  under  cost  in  the  long  run  and 
either  on  or  ahead  of  schedule) . 

4  step  approach  avoids  technical  leveling. 

GS-13  —  My  experience  is  limited  to  the  conventional 
source  selection  approach.  Therefore,  I  have  not  responded 
to  Part  II,  Sections  A-C.  However,  based  on  my  experience,  I 
believe  the  ASD  streamlined  approach,  as  outlined  in  this 
questionnaire,  is  definitely  the  best  approach.  Properly 
implemented,  this  approach  should  work  well  for  all 
competitive  acquisitions. 

GM-13  —  The  ASD  approach  appears  most  feasible  for 
small  system  buys.  The  BMO  approach  should  work  well  on  all 
buys  so  long  as  advisors  are  permitted  to  support  the  SSEB. 
Both  conventional  and  four  step  are  manpower  intensive  and 
manpower  ineffectively  and  in  some  cases,  defeat  the  planned 
scheduling  for  the  programs  which,  in  turn,  impacts  cost. 

GM-15  —  There  is  definitely  room  for  improvement  in 
the  source  selection  process.  Ideas  should  be  solicited  and 
evaluated . 


SPACE  DIVISION 

GS-14  —  Too  many  program  managers  rush  the  evaluation 
to  meet  artificial  deadlines  -  these  evaluations  should  be 
thorough  and  un-rushed. 

LT  COL  —  The  inefficiency  in  "conventional"  source 
selection  is,  I  believe,  caused  by  excessive  areas,  items, 
and  factors,  not  the  fact  that  we  have  an  SSEB/SSAC  etc. 
Also,  excessive  proposal  material  is  required  because  of 
"cultist"  requirements  for  this  or  that  kind  of  plan  to 
satisfy  "ility"  requirements  that  rarely  effect  the  actual 
choice  of  a  source.  Streamlined  approaches  improve  things 
because  they  cut  out  a  lot  of  the  excess  information. 
However,  even  greater  improvement  would  be  possible  if  a  lot 
of  the  surplus  plans,  etc.  and  the  bureaucratic  need  for 
them  in  a  proposal  were  eliminated  as  well. 


Streamlined  approaches  are  particularly  valid  for  study 
or  requirements  definition  contracts.  However,  for 
development  of  expensive  hardware,  I  believe  detailed 
proposals  and  a  more  comprehensive  evaluation  is 
appropriate.  For  that,  I  believe  the  SSEB/SSAC  organization 
is  best  even  if  cumbersome.  A  streamlined  proposal  and 
source  selection  group  almost  mandates  that  minimal 
information  is  provided  and  minimally  evaluated.  That  seems 
undesirable  in  selecting  the  best  source. 

GS-13  —  The  way  to  increasing  the  quality  of  source 
selection  as  well  as  reducing  the  time  to  select  a  source 
lies  with  the  requirements  personnel.  We  put  a  great  deal 
of  time  into  evaluating  information  that  is  not  going  to 
decide  whether  or  not  an  offeror  can  do  the  job.  We  need 
training  "BY  EXPERTS"  on  what  are  the  critical  factors  for 
making  an  award?  How  are  they  developed?  A  great  deal  of 
the  information  we  evaluate  now  should  be  relegated  to  a  go, 
no-go  decision. 

CAPT  —  The  Air  Force  seems  to  continually  drive  for 
the  "standard"  source  selection.  This  questionnaire  appears 
to  follow  that  logic.  Consideration  should  be  high  for 
allowances  by  the  program  manager  to  modify  (within  legal 
limits)  the  proceedings  to  take  into  account  situational 
concerns,  i.e.,  number  of  contractors,  program  phase, 
availability  of  SSEB  resources,  types  of  contractors,  et  al. 

GM-13  —  Utilization  of  past  performance  as  a  criteria 
for  award  should  be  mote  thoroughly  evaluated  for 
appropriate  application. 

The  organizers  of  streamlined  boards  must  be  very 
careful  in  selection  of  qualified  evaluators.  Rank  is 
insignificant  in  this  endeavor;  knowledge  and  experience  is 
paramount. 

GM-13  —  Typical  conventional  source  selections  pull 
together  50-80  government  people  to  evaluate  300-500  pages 
of  contractor  marketing  material.  Such  large  numbers  of 
people  and  pages,  I  believe,  are  used  to  give  the  appearance 
of  fairness.  Significant  time  savings  can  be  had  with  no 
loss  of  impartiality  and  little  risk  of  technical  or  cost 
"loss"  by  using  smaller  proposals  and  smaller  evaluation 
groups.  200-300  page  technical  proposals  can  often  result 
in  contractor  outlays  of  2-10  million  dollars  partly  offset 
by  B&P ,  IR&O ,  and  indirect  costs.  In  addition  contractor 
people  are  often  pulled  off  other  contracts  to  write 
proposals. 

GS-13  —  Most  needed  changes  in  Source  Selection 
process: 

Improvement  in  proposal  preparation  instructions. 


Rethinking  of  the  role  of  field  pricing  support  re: 
lunacy  of  DOD  Directive  7640.2  in  source  selections. 

COL  —  Exploration  should  be  made  into  combining  the 
advantages  of  the  Conventional,  4-Step  and  Streamlined 
methods.  Competition  should  be  made  as  "painless"  as 
possible  in  order  to  exact  its  benefits  form  both  a 
contractor  perspective  and  a  government  perspective. 

GM-13  —  At  SD,  I  have  been  exposed  to  all  kinds  of 
source  selections  small  and  large  ranging  from  missile  as 
the  SSA  to  having  AF  part  of  the  SSAC  selection  board.  At 
all  times,  page  limits  have  been  applied  on  proposals 
automatically.  Allowed  contractor  presentations  have  limited 
the  number  of  evaluators  and  used  computerized  systems.  1 
buy  complex  R&D  acquisitions  which  require  a  considerable 
amount  of  expertise.  If  we  were  limited  to  one  evaluator  per 
discipline  (1)  how  could  you  assure  expertise,  and  (2)  how 
do  you  eliminate  bias.  I  work  with  less  and  less  experienced 
officers  who  need  considerable  guidance.  If  you  pre- 
established  a  time-frame  and  make  it  mandatory,  it  does  not 
allow  for  intelligent  planning,  protest  resolution,  or  fair 
evaluations.  What  we  need  is  better  management.  That  is, 
people  trained  in  the  source  selection  process  to  speed 
things  up  and  make  intelligent  decisions. 


BALLISTIC  MISSILE  OFFICE 

GM-14  —  BMO  Streamlined  procedures  for  efforts  less 
than  AFR  70-15  thresholds  also  combine  the  SSEB  and  SSAC 
functions  as  does  the  ASD  procedure. 

COL  —  Streamline  process  should  be  used  by  all  AF 
procurements. 


ARMAMENT  DIVISION 

GM-15  —  I  believe  we  are  more  concerned  with 
procedures  than  we  are  with  content.  In  the  interest  of 
fairness,  we  say  "only  evaluate  what  is  on  the  written  page 
in  the  RFP" .  Other  knowledge  is  not  to  be  used.  I  think  this 
limits  us  in  arriving  at  our  selections.  If  I  know  what  is 
written  is  wrong/out  of  context/an  untruth  from  personnel 
knowledge,  I  ought  to  be  able  to  use  this  information,  i.e. 
in  a  recent  source  selection,  the  contractor  stated  they 
would  rent  a  specific  facility.  Even  though  I  knew  that  they 
didn't  intend  to,  I  could  only  evaluate  what  they  proposed. 

GS-12  —  The  biggest  problem  I  run  into  is  trying  to 
write  the  RFP  instructions  in  such  a  way  as  to  get  the 
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documentation  needed  for  evaluation  without  telling  the 
contractor  what  to  propose. 

GM-13  —  DCAA  audits  take  entirely  too  much  time.  When 
competition  is  present,  full  scale  audits  are  questionable. 

A  streamlined  audit  process  should  be  established  that  is 
less  time-consuming.  One  thing  that  can  be  done  is  for  the 
processing  office  to  request  audits  from  the  DCAA  instead  of 
going  through  the  ACO.  The  ACO  often  takes  a  week  to  request 
DCAA  audits.  A  copy  of  the  PCO  request  could  be  sent  to  the 
ACO.  The  ACO  could  provide  their  input  to  the  DCAA  for 
incorporation  into  the  DCAA  report  instead  of  the  present 
process  where  the  ACO  requests  the  audit  and  then 
incorporates  his  findings  into  the  final  pricing  report.  Two 
weeks  could  be  saved  because  the  ACO  waits  for  the  audit 
before  he  prepares  the  field  pricing  report. 

GM-15  —  I  believe  the  questionnaire  is  of  limited 
benefit  in  the  analysis  of  the  different  source  selection 
processes.  Many  people  do  not  like  to  participate  in  source 
selections,  therefore,  experience  factors  do  not  play  the 
appropriate  role. 


AEROMEDICAL  SYSTEMS  DIVISION 

GS-12  —  If  source  selection  team  training  were 
available  and  a  dedicated  team  were  established,  then  a  time 
limit  might  be  appropriate. 

LT  COL  —  It  currently  takes  ASD/PM  14-16  months  from 
receipt  of  our  PR  packages  to  contract  award.  They  are 
trying  to  force  us  into  AFR  80-15  source  selections  for  FSD 
programs.  Inappropriate  1  It  is  interesting  that  your  survey 
assumes  AFR  70-15  source  selection.  I  would  make  that 
assumption  for  FSD  and  Production,  too. 


AIR  FORCE  AIR  LOGISTICS  CENTER 

GS-12  —  My  experience  in  SS  has  been  to  evaluate 
against  certain  standards  set  forth  by  the  government.  These 
standards  are  poorly  written.  Major  concerns  in  the 
logistics  field-'  are  totally  avoided.  After  I  completed  my 
last  evaluation,  and  one  contractor  definitely  had  a  better 
proposal,  I  was  directed  to  change  my  evaluation  until  the 
contractors  were  equal.  Why  waste  my  time? 

GS-12  —  1.  Source  selection  has  evolved  into  a  long, 
tedious  process-  much  to  be  avoided.  The  evaluations  are 
fragmented  and  almost  avoided.  We  tend  to  use  source 
selection  to  bring  competing  contractors  up  to  an  equal 


technical  level  thru  the  contractor  inquiry  and  reports;  and 
price  then  becomes  the  only  discriminator.  Our  source 
selections  are  expensive  to  us  and  to  the  contractors. 
Therefore/  if  a  contractor  doesn't  understand  the 
requirement  we  should  go  ahead  and  cut  him  out.  We  don't  - 
we  carry  him  and  pump  him  up.  Hurts  our  credibility.  Then  we 
sometimes  end  up  with  a  contractor  that  never  did  understand 
the  requirement. 

2.  The  person  who  is  writing  the  SOW  should 
also  write  the  instructions  for  the  proposal  and  the  factors 
and  standards.  More  information  on  this  is  in  the  AFALC 
lessons  learned  library  as  a  consistent  and  continuing 
problem. 

3.  It  is  tremendously  demoralizing  when  the  SSA 
selects  because  of  politics, not  the  source  selection 
results.  A  lot  of  time  and  money  and  energy  is  wasted 
because  someone  didn't  have  the  guts  to  direct  sole  source. 

GS-12  —  I  am  not  sure  that  the  SSEB  appointing 
authority  has  the  proper  clout  to  get  the  commitment  by 
supervisors  to  provide  SSEB  people.  It  does  not  appear  that 
they  (SSEB)  are  formally  appointed  and  committed  to  the 
process. 
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